POL00029012
POL00029012
Filed on behalf of the: Defendant
Witness: Steven Paul Parker
Statement No.: First
Exhibits: SPP1
Date Made: 16 November 2018
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
IN THE HIGH COURT OF JUSTICE
QUEEN'S BENCH DIVISION
ROYAL COURTS OF JUSTICE
BETWEEN:
ALAN BATES & OTHERS
Claimant
AND
POST OFFICE LIMITED
Defendant
WITNESS STATEMENT OF STEPHEN PAUL PARKER
I, STEPHEN PAUL PARKER of Lovelace Road, Bracknell, Berkshire RG12 8SN WILL
SAY as follows:
1. I am employed by Fujitsu Services Limited (Fujitsu) as Head of Post Office
Application Support.
2. I am authorised to make this statement on behalf of Post Office Limited (Post
Office), the Defendant in these proceedings, in relation to the Horizon Issues trial
listed for March 2019.
3. The facts set out in this statement are within my own knowledge, or if they are
outside my knowledge, I have explained the source of my information or belief.
4. In this statement I refer to copy documents attached and marked Exhibit SPP1.
BACKGROUND
5. I started working on the Royal Mail Group Account, later called the Post Office
Account, in July 1997, before the introduction of Horizon. I have continued to
provide support to the Post Office Account in my various roles at Fujitsu
throughout the whole of Horizon’s life.
6. Prior to my work on the Post Office account, I held a number of roles within the
IT industry, including an in-house operations and support role for critical online
systems, support consultancy services and design and development role in an
AC_153675905_1 1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
End User Technology group in relation to Microsoft tools for internal and external
customers.
7, I began working in July 1997 as a support consultant and deputy manager within
the Software Support Centre (SSC) for the Royal Mail Group Account, providing
third line application support for the Horizon application. Within this role I was the
lead designer and part of development team for the internal website providing
support knowledge (KEL) database, technical documentation management and
operational change control. I also assisted the SSC manager in the provision of
the support service and operational management. Although I did not have the
formal title, I acted as the deputy manager to the SSC as a whole.
71 Between December 2009 and March 2010 I was a full time Problem Manager /
Operational Manager of the SSC, responsible for the management of incidents
through the whole support process.
7.2 In March 2010 I became the Manager of the SSC and was responsible for the
provision of third line application support to the Post Office Account, including
the management of the staff working on the account. The SSC subsequently
expanded into a shared service, providing support services to a number of
Fujitsu customers, the largest of which is still the Post Office Horizon system. As
head of this unit I am currently responsible for strategic support, managing 25 —
AO staff.
RICHARD ROLL'S STATEMENT
8. I have been asked to comment on the witness statement of Mr Roll dated 11 July
2016 put forward by the Claimants in relation to the Horizon Issues trial in March
2019. I worked with Mr Roll while he was employed by Fujitsu in the SSC (I
understand from Fujitsu's HR department that this was from 5 March 2001 until
17 September 2004). As explained above, although I did not have the formal
title, I acted as deputy manager while Mr Roll was there.
9. The Horizon system was rolled out across the Post Office network between 1999
and 2000. In 2010 there was a migration from the system commonly referred to
as "Legacy Horizon" to an online version ("HNG-X" or "Horizon Online").
Therefore, when Mr Roll refers to the Horizon system he is referring to Legacy
Horizon and where I respond to an assertion made by Mr Roll I am also referring
to Legacy Horizon unless otherwise stated. Much of my statement also applies to
Horizon Online, however.
AC_153675905_1 2
10.
11.
12.
13.
14.
15.
16.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
I found Mr Roll to be a conscientious worker and provided him with a personal
reference for a position that he applied for after leaving Fujitsu (it was a personal
reference because Fujitsu does not allow staff to provide references on behalf of
the company).
In his statement Mr Roll suggests that there were frequent instances of software
problems in Horizon that had an impact on branch transaction data and which
Fujitsu resolved "remotely" (i.e. not in a branch), not merely by changing the
software but also by frequently changing branch transaction data (by injecting
new transaction data and by editing or deleting existing transaction data), without
informing branches that such actions were being taken. As I explain below, those
suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is
inaccurate and misleading.
In order to explain why, it is necessary to have a proper understanding of what
Fujitsu was able to do, what the effects of its actions were and the extent to which
such actions had any effect on transaction data in branches. This is described
further below, but I would like to make some general responses immediately.
It is correct that the SSC had (and has) the ability to view data in branches and
other sources such as data centres remotely (in read-only mode): that is to be
expected given their support role which is described in paragraph 26 below.
Post Office did (and does) not have the same ability — what it could (and can) do
is view copies of transaction data as replicated from Horizon to a variety of Post
Office systems. In Horizon Online, facilities have been added which allow Post
Office to remotely examine branch data held in the Branch Database (BRDB) in
read-only mode for a variety of business reasons, such as monitoring levels of
cash that are in branches based on the data entered onto Horizon by branch
staff.
It is also correct that issues sometimes arose which necessitated changes to the
Horizon software. However, this was not something that Mr Roll played any
significant part in, as I describe in paragraphs 34 and 35 below.
It was (and is) theoretically possible for there to be a software problem which
could cause a financial impact in branch accounts, but this was (and is)
extremely rare and Horizon's countermeasures were (and are) very likely to pick
such matters up. In my experience, these problems have always represented a
very small proportion of the issues which led to software changes and a very
small proportion of the overall issues dealt with by the SSC.
AC_153675905_1 3
POL00029012
POL00029012
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
17. On the very rare occasion that a software problem which could cause a financial
impact in branch accounts arose, it would be investigated and resolved and
Fujitsu would determine its impact on the Horizon estate and inform Post Office
of any financial impact on branches so that they could be resolved.
18. In Legacy Horizon it was possible for the data in a particular counter in a branch
to become inconsistent with replicated copies, and Mr Roll appears to be
describing this in paragraph 17 of his statement. In that situation there could be
remote management by Fujitsu to correct the problem, but branch transaction
data was not changed in any way. As explained in paragraph 55 below, the work-
round involved replicating the correct data from another counter in the affected
branch or from the data centre copy.
19. The suggestion that Fujitsu edited or deleted transaction data is not correct. In
Legacy Horizon it was not possible to delete or edit messages that had been
committed to the message store. See the explanation of my colleague Torstein
Godeseth in paragraph 37 of his witness statement dated 27 September 2018
which reflects my understanding and experience of Horizon’s functionality.
20. In para 18 of his statement, Mr Roll also suggests that in Legacy Horizon he and
others in the SSC could "insert transactions and transfer money without the sub-
postmaster knowing". However:
20.1 No Fujitsu personnel have ever had the ability to "transfer money" out of Horizon
into, for example, an individual's account.
20.2 Some members of the SSC were (and some remain) able to insert transaction
data. SSC access privilege gave the ability to inject transactions, but appropriate
change controls were in place and no such insertion would have happened
without complying with those controls.
20.3 I should make it clear that, in this witness statement, I am concentrating on what
the support process is designed and able to do and not any malicious misuse of
these facilities. Malicious misuse makes most things possible, as with any other
IT system, however, Horizon had a number of checks and security settings in
place that made it very difficult to carry out malicious misuse. In any event such
misuse would have been discovered by consistency checks or colleagues (all
access was controlled and audited) and would have resulted in instant dismissal.
But even a malicious user would not have been able to “transfer money’.
21. As there is no way in which money could be taken from a branch and moved
anywhere else (for example into the employee’s bank account), it follows that
AC_153675905_1 4
21.2
22.
23.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
there was no motive for anyone to do this. It is not clear to me why anyone at
Fujitsu would insert transaction data into a branch's accounts without there being
a legitimate reason for doing so. Furthermore:
Any transaction that was inserted would immediately cause a discrepancy to
arise in the branch's accounts. For example, if a transaction were to be inserted
which stated that £1,000 of stamps had been bought by a customer who paid
cash, that would immediately cause a reduction in stock levels of stamps in that
branch and the branch would have £1,000 less in cash than Horizon expected it
to have.
In other words, although a transaction could be inserted, it would immediately
become apparent that this had been done and ultimately it would not benefit any
member of staff to behave in this way.
It is correct that the "remote access" described above could have been carried
out without the permission of a Subpostmaster. However, any additional
transactions inserted remotely would be identifiable as such from the transaction
logs that are available to Subpostmasters from Horizon.
I will now describe the structure of the Horizon support teams during the period
when Mr Roll was employed by Fujitsu before responding to the specific
suggestions that Mr Roll makes.
The Horizon Support Teams
24.
25.
26.
26.1
There were four lines of support for Horizon while Mr Roll was employed by
Fujitsu and they are described in paragraph 26 below. There are still four lines of
support for Horizon today, albeit that some names have changed and some
responsibilities have moved around teams.
It is common within the industry to have a multi-level support model. Generally,
as you move up through the levels of support the cost of the staff providing the
service increases because they are more qualified. Having said that, there is
often overlap of skills between adjacent lines of support and while a team may be
responsible for a particular level of support, staff within that team can have skills
which allow them to perform a role that is more usually performed by the next
level of support.
The four lines of support for Horizon while Mr Roll was employed were as follows:
1* line: The 1* line involved several different elements:
AC_153675905_1 5
POL00029012
POL00029012
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
26.1.1 the Horizon Service Desk (HSD)' was a helpdesk operated by Fujitsu
that branches could contact with issues relating to the Horizon
application or the hardware provided in branch by Fujitsu to run the
Horizon application. I estimate that there were around 80 members of
the HSD while Mr Roll was employed, who:-
(a) dealt with straightforward enquiries such as password issues and
scheduling engineers to attend branches to investigate reports of
hardware issues;
(b) monitored the live estate and took corrective actions defined in
knowledge documents (this role was fulfilled by two units: the
System Management Centre (SMC) for data centre events; and
the Counter Eventing Team for Post Office counter events. The
corrective actions did not involve any changes to any branch
transaction data); and
(c) referred other issues to the 2"¢ line support function.
26.1.2 there was also a 1* line Communications Management Team operated
by Fujitsu which specifically focused on communication incidents; and
26.1.3 Post Office also operated a 1* line helpdesk for operational issues
called the National Business Support Centre (NBSC).
If a branch required assistance to attempt to determine the cause of a
discrepancy they would contact NBSC in the first instance. Discrepancies are
not unusual in a retail system. They indicate a difference between the operator's
declaration of cash and stock on hand and the systems calculation and as such
are a business operation issue. However, it was not always possible for NBSC to
identify the cause of a discrepancy. For example, a user may enter a deposit of
£100 into a customer's bank account on Horizon but rather than taking £100 from
the customer, they may make a mistake and give the customer £100 as if it had
been a withdrawal. In that scenario, NBSC would not have been be able to
identify the cause of a discrepancy. Clearly, NBSC is also unable to assist when
losses have been caused by theft.
If NBSC were unable to identify the cause of a discrepancy they would often fall
back on a default statement along the lines of “this looks like a software issue"
so that the SSC would investigate it. However, Mr Roll's statement that "/iJf an
‘ The HSD has also been known as the Horizon System Helpdesk and the Horizon
Incident Team.
AC_153675905_1 6
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
error was referred to us then it was extremely unlikely to be due to a mistake
made by a postmaster" is not correct. The vast majority of discrepancies
investigated by the SSC as pseudo “software issues” were (and are) not caused
by software issues.
2" Jine: 2° line support was provided by senior members of the HSD and SMC
and junior members of the SSC (as explained below, the SSC also fulfilled a 3%
line support function). The 2" line support function mainly involved staff
searching knowledge articles based on the descriptions of issues reported by
branches, gathering evidence and applying simple, well-defined work-arounds
(often on the phone). An example of this would be resetting passwords.
3" line: the SSC also provided 3” line support. The staff that provided 3" line
support had a detailed knowledge of the Horizon application based on
documentation and some inspection of source code. They:-
26.3.1 designed, tested and documented work rounds for the 1*' and 2™ lines
of support;
26.3.2 applied analytical skills to the symptoms and evidence gathered by the
1* and 2™ line functions and undertook in-depth investigation into
incidents (incidents are the basic unit of work for the support team and
come from helpdesk calls and other Horizon support teams);
26.3.3. undertook complex configuration (configuration items can be used to
alter the behaviour of the application) and data fixes which might have
required the generation of special tooling;
26.3.4 designed, wrote and documented new support tools;
26.3.5 undertook source code examination, complex diagnosis and
documentation (including methods to recreate faults) of new
application problems before sending them to the 4" line support group
for root cause software fix; and
26.3.6 provided technical support to other internal Fujitsu teams working on
Horizon.?
The 3” line support function used a system called Peak (until 2003 it was called
PINICL) to log and manage incidents passed to them which were suspected to
? An example of this which applies to Horizon Online is support to the Management
Support Unit (MSU) who are responsible for the Reconciliation Process documented in
SVM/SDM/PRO/0020 (Reconciliation and Incident Management Joint Working
Document). The reconciliation process also applied to Legacy Horizon.
AC_153675905_1 7
POL00029012
POL00029012
27.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
be faults. It also maintained a Known Error Log (KEL) which describes the
symptoms of problems with some analysis of causes, (potential) solutions to the
problems and workarounds that might be needed before a permanent solution
can be implemented. The Peak and KEL systems are still in use today and are
described further in paragraph 62 onwards below.
4" line: 4" line support staff had an intimate knowledge of narrow areas of the
system and were (and are) ultimately responsible for the production of
permanent fixes to repair the root cause of an incident or problem in the live
application. They had knowledge of computer languages which they used to
amend source code to fix problem in the live application code. There was often
overlap between 4" line and developers, who added new features into the
application.
The structure of the support for Horizon is broadly the same today. One of the
main changes is that the HSD function is no longer operated by Fujitsu (it has
been operated by Atos since June 2014).
Further context — call volumes and transfers
28.
29.
30.
Between 1 January 2001 and 31 December 2004 (the years Mr Roll worked for
Fujitsu), the SSC received a total of 27,005 calls, meaning that on average 563
calls per month were dealt with over this 4 year period. This is shown by a
spreadsheet prepared by a team in the SSC which appears at Exhibit SPP1 (Tab
"RRP_Live_Peaks_Into_SSC"). Where (as here) I analyse data in this statement,
the analysis is mine.
Transferred calls (i.e. those not resolved by the SSC) are of interest. A very
small proportion of calls transferred to 4" line support would have concerned
software errors requiring resolution, so it would be interesting to know the number
of calls transferred to them. However, while the SSC have records of the volume
of transferred calls, we do not retain records of where they are transferred to and
it is not the case that all of these would have been transferred to 4" line support.
For example, incidents would often arrive at SSC from internal teams for routing
back to helpdesks.
As evidenced by the data in Tab "RRP Live Peaks Out of SSC" of Exhibit SPP1,
an average of 78 calls per month (14%) were transferred to teams outside SSC,
for example, to 2” and 4" line support. This indicates that only a small volume of
calls received and escalated to the SSC related to software errors requiring
resolution.
AC_153675905_1 8
POL00029012
POL00029012
31.
32.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
In terms of calls to the HSD, Fujitsu only holds records of the volume of calls
made to HSD from December 2008 onwards. I understand from my colleague
Sandie Bothick (Fujitsu's Service Delivery Manager) that the HSD received
13,225 calls in December 2008 and 13,005 in January 2009. The witness
statement of Angele Van Den Bogerd provides that Post Office’s NBSC call
volume data shows an average of 1,096 calls per day were received in the period
2015 to 2018. While these figures relate to different periods, they indicate that
only a tiny proportion of issues reported to 1% line support helplines were
escalated to SSC.
From the SSC, only a tiny proportion of incidents were escalated to the 4" line
support team. It follows that only a tiny fraction of incidents raised actually
needed to be looked at by the only team who might potentially effect changes in
software.
The SSC and Mr Roll's role within it
33.
35.
36.
I agree with Mr Roll's recollection that there were around 30 individuals working
on the 6" floor in Bracknell (i.e. in the SSC) at any one time while he was
employed.
As noted above, the SSC team provided both 2" and 3” line support. As with
any mix of people, there are (and were) various levels of talent within SSC. Mr
Roll was primarily used in Operational Business Change (OBC), which involved
supporting the engineers who were opening and closing branches and increasing
and decreasing the number of counters in branches. Mr Roll would also have
been regularly correcting the application environment after engineers had
replaced failed counter hardware and clearing temporary files to increase disk
space. This could fairly be described as 2" line work and it was done by the SSC
because it required a higher level of access to the system than other support
teams had.
Some members of the 3° line support group identified the need for software fixes
via source code examination and would pass this on to the 4" line team for a
code fix to be written. Mr Roll did not play any significant part in this and was not
involved in any extensive source code examination. An application code fix
would not be written by anyone in the 3” line team and he was not involved in the
provision of 4" line support
I disagree with Mr Roll's suggestion that much of the work being carried out by
the SSC while he was employed could be described “as "fire fighting" coding
problems in the Horizon system." There were times when the SSC was very
AC_153675905_1 9
POL00029012
POL00029012
37.
38.
39.
40.
40.1
40.2
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
busy, for example, networking problems causing application issues across the
whole estate and data centre outages. But there were only rare circumstances
where a coding issue had an estate wide impact and, in those instances, Mr Roll
would have been involved executing avoidance actions to mitigate impact to the
estate (i.e. following established work-arounds) rather than working on the root
cause.
The SSC had (and has) access to view, but not amend, source code. Senior
members of the team would have looked at it from time to time to confirm
exactly how the Horizon application would process a given input and what the
outputs would be when investigating specific issues or general education on how
the system works. However, the access was rarely used. Moreover, Mr Roll was
not working at a level where he would be required to review code. His role in the
SSC was predominantly following work around processes designed by other
people and making configuration changes.
In support of the statements above relating to SSC workload I have undertaken
some analysis of the work carried out by SSC between 1 January 2001 and 31
December 2004 (as stated above Mr Roll was employed from 5 March 2001 until
17 September 2004). My colleague John Simpkins provided the summary data
from the Peak system, which I analysed.
When an incident is resolved, the SSC team member (or technician as they are
sometimes called) types a summary of the incident (known as a Final Response)
and allocates a response code to the incident in order to classify it. While
guidance is provided on when to use each response code (see paragraph 64.5
below), allocation is the subjective view of the technician closing the incident and
there is no re-examination of the response codes later to ensure consistency.
With that in mind, the final response codes that were allocated to incidents (i.e.
Peaks) reported to SSC between 1 January 2010 and 31 December 2004 were as
follows:-
known issue / work around - 35.3%;
admin - 27.5%
reconciliation - 15.7%;
potential user error - 10.9%;
potential software error - 8.3%; and
AC_153675905_1 10
POL00029012
POL00029012
40.6
41.
41.2
41.3
41.4
42.
42.1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
hardware error - 1.2%.
It should be noted that:-
a major part of 1* line’s raison d’étre is to deal with user error and therefore the
percentage of issues attributable to user error would be much higher at 1* line;
very few hardware incidents reached the SSC because they were the preserve of
the HSD (i.e. they were relatively easy to spot and therefore filtered out by 1%
line support);
8.3% of calls to the SSC (2252) are attributed to potential software errors over
these four years. This includes duplicates and does not provide any clarity on the
significance of the error corrected. Many software errors, particularly in a new
product, were insignificant, such as correcting capitalisation in printed output. I
cannot be more precise without examining each Peak and even then it might not
be possible to determine how a Peak should have been properly classified
(Peaks are essentially notes made by Fujitsu personnel to chart the progress
made in resolving an issue and these notes can vary in fullness and clarity); and
Classifying an incident as a "potential" software error does not necessarily mean
that there was a software error and, even if there was, it does not mean that the
error was one that could have caused a financial impact in a branch's accounts
(a large proportion of these would be errors in numerous data centre resident
systems that the Subpostmaster never sees — errors were often as trivial as the
use of “Kg” instead of “kg” on receipts). As stated in paragraph 16 above, such
errors were extremely rare. They were all resolved (resolutions include a source
code fix, a configuration fix or a workaround).
Mr Roll's suggestion in paragraph 10 that software issues in Horizon "routinely"
caused discrepancies in branch accounts is misleading. In the vast majority of
cases such an occurrence would cause a receipts and payments (R&P) mismatch
that would be flagged by the branch system as part of the balancing process (the
Horizon system carries out self-consistency checks which generate alerts in the
event of a receipts and payments mismatch that are picked up by SMC and
incidents raised for the SSC) and appear on MSU reporting. These would then
be investigated and resolved by the SSC.
Since the introduction of Horizon in 1999 there have been 735 live incidents
which refer to “Payments and Receipt mismatch’ (i.e. incidents recorded against
components of the system providing Horizon service to Post office rather than
incidents raised against test systems). This figure has been obtained using a
AC_153675905_1 11
POL00029012
POL00029012
42.2
42.3
43,
45.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
textual search across all incidents where the title or one of the incident updates
contains all of the words “receipts”, “payments”, “mismatch”. It should be noted
that this is not 735 unique incidents; there will be a lot of duplicates with the
same root cause. The only way to determine how many unique incidents there
were would be to manually review all of the incidents. All of them were resolved.
These incidents are reported as a result of self-consistency checks carried out by
Horizon. It should be noted that a R&P mismatch is not only caused by a
software error. It can also be caused by incorrect product reference
(configuration) data
Receipts and payments mismatches happened more often during the early life of
Horizon (see Tab “All RnP by RCode and Date” of Exhibit SPP1). My analysis of
that data shows that there were around 8.6 such incidents per month on average
between 1 January 2001 and 31 December 2004 (417 out of a total of 27,005
incidents into SSC or 1.5% of SSC incidents during that period).
Mr Roll refers to a "perception...that the Service Level Agreements between Post
Office Ltd and Fujitsu involved financial penalties payable by Fujitsu to Post
Office" (paragraph 12). I am aware that there were Service Level Agreements for
issues such as stuck transactions (Fujitsu had 10 days to retrieve transactions
that had not replicated from a counter). It is quite normal for contracts such as
the one between Fujitsu and Post Office relating to Horizon to have such
agreements. The same level of diligence was (and is) applied to all incidents,
whether an SLA was relevant of not. The possibility of financial penalties was
never a factor for the SSC.
I do not understand what Mr Roll means when he says that “any discrepancy in
the post office accounts had to be resolved speedily" (paragraph 12). There was
(and is) a process run by the Management Support Unit (MSU) which involves
examination of various system reporting and may result in Business Incident
Management Service (BIMS) entries going to Post Office. An incident may also
be raised by MSU with the SSC to provide support to the MSU in resolution of the
BIMS. These are subject to Service Level Agreements and Mr Roll may be
referring to this process. However, if Mr Roll is suggesting that Fujitsu routinely
rushed out fixes or workarounds due to SLA time pressure, that is not the case.
Fixes would be expedited based on service impact. It would be quite wrong to
suggest that they were not done properly because of any SLAs.
It is correct that there are a limited number of opportunities to release software
updates and that these releases could take 6 weeks or more to be released to the
live service (Mr Roll's statement, paragraph 13). These longer timescales would
AC_153675905_1 12
POL00029012
POL00029012
46.
47.
47.2
47.3
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
be employed for non-urgent updates wrapped up into a consistent set for
deployment. On those rare occasions when a problem has an impact on the
financial integrity of the system a "hot fix" would be deployed which involved
much shorter timescales. I would expect a timescale measured in days not weeks
to deploy a hot-fix. Mr Roll also states that a bug could reappear several weeks
after a coding fix had been released due to software issues. I am aware of only
One or two cases where a fix regressed in my time at Fujitsu.
In paragraph 14 Mr Roll states, "/ would reiterate that the main recurring issues
were software issues." It is a symptom of working within a software support team
that the majority of issues that come in have been attributed to a software issue
by, for example, a lower line of support. This can lead to a mind set of “look at all
these Horizon errors”, but what this indicates to me is that the previous levels of
support are functioning correctly, removing the majority of other causes (user /
hardware problems). It does not indicate that the majority of Horizon errors could
be attributed to software.
Mr Roll states (paragraph 7) that "[sjoftware programs were written by us to strip-
out irrelevant data, to enable us to more easily locate the error.". The support
tools are used to filter information and present information to technicians in ways
that make the support process easier. I am aware of two support tools (also
known as software programs) that were written while Mr Roll was employed by
Fujitsu:-
the Smiley support tool written by my colleague John Simpkins, which
amalgamates information from various sources (e.g. databases) into a single
view pertinent to a particular support task and provides a unified interface to run
various tools to achieve a single support outcome; and
another tool (I cannot remember its name) written by my colleague Richard
Coleman whose function was to retrieve messages (i.e. data) from the
correspondence server to local text files for examination and which was
eventually subsumed into the Smiley support tool.
Neither of these tools changed the underlying data in a branch's accounts.
The work carried out by Mr Roll
48.
Mr Roll states that he would investigate financial discrepancies that had arisen in
branches by “work[ing] sequentially through all transactions over the relevant
period, and also work[ing] through thousands of lines of computer coding"
(paragraph 7).
AC_153675905_1 13
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
48.1 It is not the role of the SSC to routinely investigate discrepancies.
48.2 In very rare circumstances a discrepancy could be caused by a software issue
and in those circumstances it might be necessary for the 4" line support function
to work through thousands of lines of computer coding to determine the root
cause. However, Mr Roll would not have worked through thousands of lines of
computer coding to investigate a discrepancy in a branch: he did not work in the
4" line.
49. Mr Roll further states that if SSC was "unable to find the cause of the discrepancy
then this was reported up the chain and it was assumed that the postmaster was
to blame" (paragraph 10). That is not my experience: it is a simple truth of
support that the majority of issues reported in a system are attributable to user
action or user misunderstanding of system functionality.
Hence, someone
working in a support environment analysing a new issue would examine the
possibilities of user error as a first hypothesis but any final conclusion is only
generated based on the evidence. Where the evidence does not support a
conclusion that there is a problem with Horizon, the SSC feeds the existent
factual data back to Post Office and might say something along the lines of "all
indications are that the branch has made a mistake", but Fujitsu neither attributes
"blame" or agrees the final conclusion with the Postmaster
50. When an incident is resolved, the SSC team member (or technician as they are
sometimes called) types a summary of the incident (known as a Final Response)
and allocates a response code to the incident in order to classify it. While
guidance provided on when to use each response code, allocation is the
subjective view of the technician closing the incident and there is no re-
examination of the response codes later to ensure consistency.
51. The Peaks that Mr Roll works on while employed by Fujitsu indicate that he dealt
with 915 incidents (see Tab "Final Responses" of Exhibit SPP1). To give some
clarity on these incidents I have analysed their final response codes that were
allocated to them by Mr Roll. He classified them as follows:-
Response code % allocated
by Mr Roll
Known issue / work around 61.9
Reconciliation 14.5
Admin 9.3
AC_153675905_1
14
POL00029012
POL00029012
82.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Potential software error 3.2
This supports my recollection that Mr Roll mainly followed work-arounds devised
by other people (61.9%) and that he was rarely involved in the detailed
examination of potential software errors (3.2%). As explained in paragraph 41.4
above ‘potential’ software errors do not necessarily mean software errors, let
alone software errors resulting in discrepancies to branch accounts.
Remote access
53.
55.
I understand from Post Office's solicitors that the term "remote access" has been
used to describe Fujitsu and Post Office carrying out the following actions when
not physically present in a branch:-
accessing branch data in read-only mode;
inserting new transaction data; and
editing or deleting existing transaction data.
As noted above, it is correct that Fujitsu had (and has) the ability to view data in
branches remotely given their support role.
Mr Roll claims that "fdjuring the course of resolving the software issues, we
would frequently access a Post Office counter IT system remotely" (paragraph
15). Mr Roll gives the example of ("when a binary bit would "flip", thus a "1"
became a "0"):~
This probably relates to a condition known as a CRC (Cyclic Redundancy Check)
error which would happen when a hard drive became faulty at a branch counter.
Although this is a hardware problem, remedial action was needed by the SSC to
resolve. To clarify, this process did not involve changing any transaction data.
As explained by Mr Godeseth in paragraph 35 of his statement, in Legacy
Horizon:-
55.2.1 all counter data was held in a bespoke message store in each branch®
and the data was replicated to all counter positions in the branch and
from the branch to the data centres where it was held in
correspondence server message stores;
3 A message means data and transaction data is a subset of the data in the message
store.
AC_153675905_1 15
POL00029012
POL00029012
56.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
55.2.2 any data inserted into the message store at the data centre (for
example reference data or authorisations for banking transactions)
would be replicated back to the branch counters; and
55.2.3 selected data was extracted from the correspondence servers to
update Post Office's back end systems.
If one of the sets of data on a branch counter became corrupted it would
generate an event that would be picked up by the SMC and/or reported to HSD
by the branch (an incident reporting a “CRC Error”). There were a total of 629
CRC errors over the life of Legacy Horizon (see Tab "All CRC by Date" of
Exhibit SPP1).
The issue would be reported to the SSC, who would delete the entire set of data
on that counter and replace it with a copy of the data from one of the other
sources that had not been corrupted. While this process involves deleting and
replacing a set of data, no new data is produced; all that happens is that the
replicated data is used to replace the data that has become corrupted from
another counter in the branch. It would have been necessary for the SSC to
inform a branch before carrying out this task because it is likely that any attempt
to use that counter would fail while the process was being carried out. In Mr
Roll’s capacity as an OBC specialist, he would have been involved in this type of
activity.
Alternatively, Mr Roll's reference to a binary code flipping may relate to a
configuration item (an item that can change the behaviour of the application)
which can become locked in the wrong binary setting (1, 0). One example of
which would be a stock unit lock which, in the wrong state, would prevent
updates to stock units within a branch. This issue was corrected by a member of
the SSC accessing Horizon remotely, but it did not involve accessing or editing
transaction data in any way or re-creating databases.
I cannot think of any other examples of incidents that Mr Roll may be referring to
in paragraph 15.
Mr Roll claims that "some errors were corrected remotely without the sub-
postmaster being aware" (paragraph 16).
It may be that Mr Roll is referring to issues relating to the end of day concept in
Legacy Horizon. Essentially there was a cut-off point for transactions every day
at 7:00pm and each counter had to write an end of day message to the branch's
master counter to enable the master counter to write a branch end of day
AC_153675905_1 16
POL00029012
POL00029012
57.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
message, which would then trigger the data centre to harvest messages
(including details of transactions) to Post Office's back end systems.
Occasionally a counter in a branch would fail to write an end of day message and
there was a process for correcting this. The issue would be reported to SSC by
way of an incident (either as a result of a call to HSD or sometimes Fujitsu could
spot issues via system events).
In lay terms, SSC would force the counter to generate a report based on the data
already in the counter. A message injected in this way would go into the audit
trail. This would not alter the branch's transaction data.
Mr Roll also states that, there were some errors where it was necessary to
“download and correct the data and prepare it for uploading back on to the post
office computer, then call the postmaster to inform him that there was problem
and that we needed two or three minutes to correct it" (paragraph 17).
57.1 It is not clear what errors Mr Roll is referring to or how he says they
were corrected. The issue referred to could be another instance
where the work round of re-creating transaction data from a
replicated copy was required as described in paragraphs 55.3 and
55.4 above.
SUMMARY
58.
In summary, the suggestion that Fujitsu would manipulate a branch's transaction
data in a way which was detrimental to a particular Postmaster and undetectable
is wrong.
All support action taken by Fujitsu is directed to ensuring that legitimate
transactions entered by Subpostmasters are correctly processed by the Horizon
application and correctly passed to other POL systems as appropriate.
Any financial corrections required are communicated by Fujitsu to Post Office for
execution or approval
Any support intervention that requires the insertion of a transaction is identifiable
in the audit trail.
There is no financial incentive for a support technician to circumvent the controls
within the system.
There are strong controls in place to prevent intervention by support staff with
malicious intent or misguided attempts at financial gain.
AC_153675905_1 17
POL00029012
POL00029012
59.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The statement that issues with coding in the Horizon system were extensive and
impacted branch finances is incorrect for the reasons stated above.
KELS AND PEAKS:
60.
61.
I have been asked by Post Office's solicitors to provide some more information
on the two toolsets used to support the Horizon system, the KEL knowledge base
(AKA KEL) and the incident management system (PINICL / Peak). This
information is applicable to both Legacy Horizon and Horizon Online except
where explicitly stated otherwise.
Fujitsu use a custom solution, developed and administrated by the SSC, which
allows us to record support knowledge into a structure known as a KEL (Known
Error Log). KELs record support knowledge which is intended to assist staff in
the support and understanding of the Horizon system. KELs do not contain the
history of an incident (see my analysis of the Peak system below). KELs are
generated for a number of reasons, for example: to explain system behaviour or
messages that originate from central and counter systems; to record symptoms
and outcomes from incidents referred by help desks or identified as a result of
Fujitsu's reconciliation processes; and to record information on issues seen in test
environments (resolved before the feature is passed on to users).
The acronym KEL is a misnomer inherited from a previous system. KEL entries
are support knowledge entries and do not have a one to one relationship with
errors on the system. There are a lot more general supporting knowledge KELs
than KELs relating to specific errors.
Guidelines for the generation and use of KELs are documented in
SVM/SDM/PRO/0875 (End to End Support Strategy) section 11 Knowledge Base
Maintenance. Although some aspects of this document need revision due to
changes in the support structure for Horizon, Section 11 is still fundamentally
correct in relation to Horizon.
KELs reflect community sourced knowledge to assist staff involved in the
support of the Horizon solution. There are no mandated rules for when a KEL
should be created other than a desire to make resolving a problem easier for all
concerned. Guidelines exist in SVM/SDM/PRO/0875. Any KELs created or
updated are referred to the SSC for approval to provide a basic check that the
information at the time of the change is valid.
KELs can be created by the senior people in HSD (2"4 line) to supplement their
own knowledge base (rather than taking an active role in the KEL process) and
AC_153675905_1 18
POL00029012
POL00029012
61.5
61.6
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
the SMC monitoring team. The 1* line helpdesk function do not create KELs
(HSD 1* line used their own knowledge base).
AKEL is a living document that reflects support knowledge at a given point in
time. KELs are not designed to provide a history of a particular symptom or
support process and a particular KEL cannot be considered the definitive source
containing all possible information regarding the problem it addresses. It is up to
the technician to check other potentially relevant KELs and information sources
(€.g. support guides) and use their analytical skills to determine the right course
of action to take in a given situation.
There have been historic requests to remove large numbers of KELs based on
date updated to reduce the number of search results that are returned to help
desks when naive search terms were used. The dates given in the KELs, while
an indicator of potentially irrelevant support information, are not precise. This led
to the concept of “deactivated” KELs (deactivated KELs are removed from the
default searches support people use although the user interface allows the user
to explicitly request a search to include deactivated KELs). At the time of
writing, there are 113 deactivated Legacy Horizon KELs and 1024 deactivated
Horizon Online KELs.
For most of its lifetime, there has been no fixed routine for the review of KELs.
If a technician recognises an inaccurate KEL as they analyse information they
are expected to update in order to improve the knowledge base. Approximately 2
years ago the KEL review forum was introduced. This forum meets weekly to
review KELs and update or deactivate as appropriate.
Before creating a KEL there is an expectation that the creator will search the
existing information to ensure that it is not duplicated. Because people express
information using different words, it is not possible for the system to perform
such a check. Human fallibility and unique expression mean that duplicated
information is present in the KEL system.
Archiving: There is no requirement to keep historic support information. Once an
item is no longer relevant to current systems it can be removed without any
implications to the support of the system. KELs are deleted when they have no
value to the support of the current systems. This can happen at any time and is
carried out by the SSC who are the final arbiters of what information is currently
relevant. That does not mean that all current KELs are still relevant; it may be
that irrelevant KELs have not been deleted yet. At the time of writing there are
1,491 deleted KELs.
AC_153675905_1 19
POL00029012
POL00029012
61.10
62.
62.1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL entries have a single field to record an incident (Peak) reference. This is not
a record of all incidents the KEL is relevant to. Generally, it is used to record the
Peak that was being investigated when the KEL was created or updated. This is
a manual entry and is not checked by the system because a KEL may not have
an associated incident. There is no requirement to update the KEL when
information in it is re-used to provide guidance on a different incident.
Peak (not an acronym) is browser-based software incident and problem
management system used by Post Office Account for all development, test and
support teams except the 1 line help desk. It enables details of the incident and
diagnostic progress to be captured in a searchable format and allows the tracking
of problems from detection through to resolution. Peak was developed in-house
by the SSC from the PinICL system it replaced in 2003. The system has been
customised and enhanced over its 15 years of operation and still continues to be
developed to Post Office Account requirements today.
When Peak was implemented, data from PINICL was migrated to Peak.
One source of Peak incidents are the 1* line support teams (including the HSD
the NBSC helplines). Peak is also used to process incidents generated by other
support units, monitoring and testing teams.
The structure of a Peak enforces a workflow (it gives a process structure leading
to a defined outcome). As a result, the Peak system has also been used to
record and progress other items loosely associated with incident management.
For example, Release management process, reference data delivery process,
Post Office Data Gateway route definition process. So the Peak system contains
incidents which do not directly impact the Horizon service provided to the Post
Office counters (AKA “Live service”). These additional types of “incident” are
differentiated by the Peak type: L = Live service, R = Release management etc
For most of its life cycle a Peak is assigned to a particular support team and a
person within that team who is responsible for the next action on the incident. As
the incident is progressed by various members of the support community, they
add textual comments and supporting evidence to the Peak to document the
progress of the incident. These updates are date / time stamped and form a
record of the diagnostic and resolution process. Progress updates cannot be
deleted / amended by users once committed to Peak. A Peak may also be
transferred between teams and people as it progresses to final resolution.
A final response code (numeric) is applied to an incident when it has reached its
conclusion along with text that supports the response code. Response codes are
AC_153675905_1 20
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
the subjective opinion of the person closing the incident and are not subject to
review but they remain the best way to classify the final outcome of an incident.
Response codes and their expected usage are documented in
SVM/SDM/PRO/0875 (End to End Support Strategy) in Section 9 Incident
closure categories. Although some aspects of this document need revision due
to changes in the support structure for Horizon, Section 9 is still fundamentally
correct and is relevant to Horizon (Legacy and Online).
The Peak system has no archiving policy. Effective incident management
requires that you can track the current issues and those from the recent past.
Data retention on Peak has been impacted by a lack of disk space, the primary
cause being large evidence files attached to Peaks. I am told by my colleague
who maintains the Peak system that:-
62.6.1 encrypted evidence files are removed one month after the incident is
closed; and
62.6.2 due to disk space issues in the past it has been necessary to remove
evidence attached to older Peaks.
KEL references can be added to a Peak entry. There is no system check to
ensure a KEL has been added since KELs are not relevant to all incidents being
processed.
Incident priorities and appropriate usage are described in SVM/SDM/PRO/0875
(End to End Support Strategy) in Section 7 and SVM/SDM/PRO/0018 (Incident
Management Process). Incidents with a financial impact on branches are treated
as high priority.
Target times to resolve software incidents are described in SVM/SDM/PRO/0875,
(End to End Support Strategy) in Section 8.
KEL ANALYSIS
MR COYNE'S KELS
63.
I have been shown paragraph 5.114 of Mr Coyne’s report, in which he says that
he has analysed 5,114 KELs to determine the scope and impact of potential
Peaks. Mr Coyne explains that out of these 5,114 KELs, he believes he has
found 163 that could be of "significant interest" and that he has referred to 76 of
these in his report.
AC_153675905_1 21
POL00029012
POL00029012
65.
66.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Post Office's solicitors have reviewed Mr Coyne's report and have provided me
with a list of KELs that they have identified as being referred to in the report
(Coyne KELs). I do not know why there appears to be a difference between this
number and the number of 76 quoted by Mr Coyne.
It is not clear what Mr Coyne means by “significant interest’. \t may be that he
means that a KEL presents evidence of a bug, error or defect in Horizon that has
caused a financial discrepancy in branch accounts. I refer to this as "financial
impact" as shorthand in this statement.
KELs are written by and for members of Fujitsu’s support groups (i.e. all of the
teams who support the Horizon solution for Fujitsu) who have a deep knowledge
of the design and operation of Horizon, and they are often expressed in a
shorthand way. I believe that it would be helpful to explain the significance and
implications of the Coyne KELs. Annexed to this statement is a table which
contains the initial explanations that have been produced by a team from SSC at
my request in the time available.
DR WORDEN'S KELS
67.
68.
Dr Worden has selected a sample of 48 KELs. A list of these KELs was passed
to Fujitsu by Post Office's solicitors (the Worden KELs).
A table which explains the Worden KELs is annexed to this statement. As with
the Coyne KELs, given the limited time available to prepare this statement, the
initial explanations have been produced by the team referred to in paragraph 66
above.
STATEMENT OF TRUTH
I believe that the facts stated in this witness statement are true.
Signed:
Date:
AC_153675905_1 22
POL00029012
POL00029012
POL00029012
POL00029012
Filed on behalf of the: Defendant
Witness: Steven Paul Parker
Statement No.: First
Exhibits: SPP1
Date Made: 16 November 2018
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Appendix 1
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
wrightm Payments 56-511 I This issue is reported as causing I This issue, referred to as the ‘Payment Mismatch’ by Mr Coyne, is dealt with I The issue caused a receipts and
33145J Mismatch discrepancies showing at the Horizon I substantively in the second witness statement of Torstein Olav Godeseth. payments mismatch in the accounts of
counter which disappeared when affected branches, which was detected
branches followed certain process by the monitoring of system events by
steps. While a workaround was Fujitsu. Post Office were informed
established by KEL wright 33145, it and corrected the relevant branch
is not clear how many corrections accounts.
were required to fix all instances of
this or event that all instances were
fixed
AC_153675905_1 1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
acha1233J
Unexplained
Discrepancies
(cash
declarations)
§.21-5.23
Discrepancies between the branch
cash declarations and the amount
received by SAP (the cash
management system). The KEL
states that this is not a user error or
anything that can be corrected at
branch level. This is therefore
consistent with the problem being
due to the existence of a software
bug
The cash planning system calculates the branch's cash levels figures from
one day to the next. This was a timing issue due to figures from a previous
day being used in association with other figures from the current day.
This issue affected the figures being used by the back end cash planning
system and did not affect any branch accounts. The KEL exists to deal with
further complaints.
In the event that there is a discrepancy between the amount received by the
Post Office cash centre in a cash pouch returned by a branch and the
amount the branch entered on Horizon as having been put in, this would be
identified by Post Office and a Transaction Correction for the appropriate
amount would be issued
No impact.
AC_153675905_1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
2
achat717T
Unexplained
Discrepancies
(cash
declarations)
5.22
There is an acknowledgment that
cash declaration discrepancies could
be due to an "Unknown System
Problem’.
By way of context, this is a generic KEL relating to the handling of calls
concerning discrepancies. There are a number of possible causes for
unexplained discrepancies between the cash declaration and figures on SAP,
including:
© Subpostmaster has made an incorrect declaration
* Transactions as recorded on the system do not match what actually
happened at the branch
© Outstanding recovery
* Withdrawn products
* Known system problem (these should be monitored for or be easy to
spot from events etc.)
* Unknown system problem
This term "unknown system problem” is a term that the KEL creator would
have used to indicate that the issue may have been caused by a new
(previously undetected) defect, that would follow the normal diagnosis
process and will then become a known system problem once fixed.
The relevant Peak (PC0202239) relates to an investigation surrounding the
possible explanation of a £240 discrepancy. The investigation concluded that
the discrepancy was likely to have been caused by human error and it
appears that the Postmaster accepted this conclusion.
None.
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
4] achaé2iP Unexplained 523 Evidence of cash declaration This issue, also referred to as the ‘Dalmeliington’ issue Is dealt with This issue caused a discrepancy in
Discrepancies discrepancies arising from clerks ‘substantively in the second witness statement of Torstein Olav Godeseth. the Subpostmaster’s outreach branch
(duplicate Rem duplicating Rem In transactions as a I Fujitsu identified the branches affected by this issue and gave the information I which was easy to identify from the
In) result of wrong messages being to Post Office to resolve the issue with affected branches. transaction logs available through
presented on the Horizon counter Horizon and the fact that separate
screen. receipts were printed for each
transaction. Post Office issued
Transaction Corrections or advised
‘Subpostmasters how to take corrective
action to remove the discrepancies.
5.I LKiang3014S I Unexplained 524 Tssue following multiple cash This KEL relates to the next KEL (MScardifield2219S) as follows: This issue may have had a temporary
Discrepancies
(multiple cash
declarations)
declarations and trial balance report
being inaccurate. On this occasion
the support department was unable to
identify the root cause of the
discrepancy although it was reported
that a correction could be made at the
Post Office counter level by redoing
the cash declaration using the same
amount already declared. Within KEL
MScardifield 22198 Fujitsu identify
the underlying software bug as being
caused by ‘cached data’ not being
updated via Riposte.
* LKiang3014S describes the symptoms of the issue; and
+ MScardifield describes the cause of the issue.
This was a software/ environmental issue involving the Horizon system
struggling to find cash declarations, which would tend to happen if a
Subpostmaster was undertaking multiple declarations in a stock unit.
This particular instance involved Riposte failing to find one of the Cash
Declarations and thus generating a temporary discrepancy. If left unresolved
this would result in a loss to the Postmaster for this period. However, when
the Subpostmaster subsequently declared the cash held in branch
accurately, this would generate an equal and opposite discrepancy that would
cancel out the earlier discrepancy.
This issue was not resolved, but further diagnostics were added to enable
further investigation should the problem happen again
financial impact, but it would have
been resolved when the branch next
declared the cash held in the branch
accurately.
AC_153675905_1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
2
MScardifield2
2198
Cached Data
Delay
5.24
Delay in ‘cached data’ being updated
via Riposte. This resulted in incorrect
data being presented in any
discrepancy, variance and balance
reports.
Mr Coyne comments that this is likely
to be confusing for the Postmaster
and could lead them to making
unnecessary modifications if they are
unaware that the problem should
clear itself overnight.
This KEL relates to the previous KEL (LKiang3014S) as follows:
© LKiang3014S describes the symptoms of the issue; and
+ MScardifield describes the cause of the issue.
This is not a'Live' issue, but something found on a Test rig. Experiments did
find that this could occur around once in 100,000 balance calls when the rig
was heavily loaded.
13 Live Peaks reference this KEL, each Peak was treated separately and the
issue was discussed with the Subpostmasters. The work around of
subsequently correctly declaring cash declarations was identified and
communicated to Subpostmasters.
As above.
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
DSeddon5426
Pp
Cash Pouch
Delivery
5.25
A failure in pouch delivery resulted in
a cash gain when the branch
declaration was carried out.
The cause of this was that the system was in “Price shopping mode” which
can be used to buy, for example, £15 of 1st class stamps. Price shopping is
not supported for products like cash, so when the branch attempted to remit
the cash deliver in while in Price shopping mode the cash remittance failed.
In this particular KEL, the remittance was repeated later and was successful
A fix to ignore “Price shopping mode" for Rems was applied to Live in April
2007.
This had a financial impact, but when
the branch investigated the
discrepancy the failed remittance
would have been identifiable in the logs
that are available to users of Horizon.
In addition:-
+ acritical event was written
that will have been picked
up by Fujitsu's support
teams and Fujitsu will have
advised Post Office to
contact the branch; and
+ Post Office's own
reconciliation procedures
would have identified that
the remittance in was not
completed successfully and
contacted the branch and
resolved the matter with a
Transaction Correction.
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
&] _ achat94L Automated 5.26 Problem affecting around 15% of ‘Automated cash declarations from self service kiosks failed at random. No impact on branch accounts.
Cash kiosk branches that prevented these I Analysis shows that this was due to rounding errors on the arithmetic carried
Declarations branches from being able to out due to use of incorrect data types. The business impact of the error is on
automatically make cash cash management and the delivery of
declarations. The lack of cash declarations have no impact on Branch accounts, but will I cash meaning the cash needed at a
result in cash planning potentially having incorrect or missing figures and branch may be incorrectly estimated,
thus failing to calculate accurate amounts of cash to send to the branch. leading to late or insufficient cash
deliveries.
This was a coding error and was fixed in December 2015
The vast majority of self service kiosks
are located in Crown branches and the
others are located in large mains
branches.
9.I DSeddon314 I Reference Data 5.30 Examples have been found in KEL’ _I The particular KEL related to the incorrect maximum values being set upin I No impact - it was not possible to rem
Q Peak records that indicate Reference I Reference Data and the counter not handling this error correctly. This was _I in the coin and therefore it could not be
Data has an impact upon daily caused by a human error by Post Office in the Reference Data. Specifically, I sold by branches.
counter activities it was not possible to rem in a particular commemorative coin because it did
not have the requisite Reference Data.
The Reference Data was fixed on 14/3/06 and a code fix to handle the
scenario better was applied in June 2006. This did affect a number of
branches during the day that the problem was live, but a message was sent
out to all branches advising them of the issue and how to correct it
AC_153675905_1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
TO] johnbascoG52 531 ‘An Automated Payment transaction _ I For some unknown reason, the counter was associating an Automated No impact.
22H was reported as having failed due to I Payment transaction with an invalid client code. There was no fault in the
“Unknown Agent Code 3046”. Client I Reference Data. The fault was impossible to reproduce, and so not
account code 3046 was found tonot I understood.
exist in Reference Data and the fault
was not reproducible when the Failure to complete a transaction would not produce an error in accounts - a
problem was analysed and tested. It I double entry transaction would either all succeed or all fail
was acknowledged that, due to the
business impact, a fix would be
provided to check and validate the
client account code exists in
Reference Data before the
transaction is committed.
Ti] achatoL Reference Data 5.32 KEL documents how branches were _I Incorrect Reference Data was issued, which had the effect that payments of I No impact.
Errors unable to accept cards for rent and —_I Council Tax to Vale of Glamorgan Council were rejected. The Reference
council tax payments due to incorrect I Data was fixed overnight and it all worked the next day. Failure to complete a
Reference Data. transaction would not produce an error in accounts - a double entry
transaction would either all succeed or all fail.
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
2] MWrighti458
Q
Withdrawn,
Products
5.33
Tssue involving withdrawn products
impacting a branch accounting
position because the Postmaster will
have products that cannot be
accounted for as there is no
remaining Reference Data for them to
later declare that stock item in the
accounts,
When products are withdrawn, there is a withdrawal process which requires,
users to have Remmed Out the relevant products (which involved the stock
being returned to the stock centres) or the stock would need to be adjusted
The grace period for the Subpostmaster either Remming Out or adjusting the
stock was 6 weeks. After the Reference Data was withdrawn, any unused
stock would result in a discrepancy (a change was introduced by Impact in
205 which meant that this could no longer occur). The assumption was that
the branch had actually sold the missing stock and failed to record it on
Horizon so the stock was removed from the system and a cash discrepancy
would be generated for its value
In this particular case, the branch failed to Rem Out its unused stock of
obsolete products (stamps) within the grace period despite being asked to do
so on multiple occasions, This left the Subpostmaster with a problem she
could not solve herself, but needed Post Office support to do so. To resolve
the issue, corrective transactions were added to the Message store in
consultation with Post Office at the data centre to reflect the value of the
obsolete stock which enabled the branch to roll over correctly.
Temporary financial impact caused by
the Subpostmaster's failure to follow
the correct procedure in relation to the
Remming Out of obsolete stock
Fujitsu was able to resolve the issue
by adding a compensating transaction.
AC_153675905_1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
T3]__wbra5353J_I Reference Data 5.34 The customer was charged twice for I This was a fault involving a self-service kiosk, which resulted in a customer I No impact.
Errors the same transaction which was being debited three times, after which the session was cancelled
reported to be a side effect of errors
within Reference Data. This was an issue with how the NIGR kiosk interfaced to Horizon, which
attempted multiple Debit Card transactions with the same ID. NOR
diagnosed this as being due to some invalid Reference Data being sent to the
kiosk.
The fault appears to result from two causes:
i, faulty Reference Data (the result of human error), which was easily
corrected
ji, a fault in the kiosk software, which came from an external supplier
14] GMaxwell3651 Duplicate 5.38 Failures or interruptions in service This was an issue where approximately 835 transactions were sent to I No impact.
K Payment with the harvesting process can Streamline (Post Office's previous payment services provider) on 2
Transactions cause duplicate payment successive days due to an operational issue in Horizon back end processing.
transactions to be processed.
These were identified by Streamline as duplicates and not processed so
customers were not charged twice. This also resulted in errors picked up by
Fujitsu's Reconciliation service so it was trapped by both ends of the
interface. The DRS (Data Reconciliation service) takes inputs from multiple
sources (counters, TPS database and Fl reports) and compares them and
produces reports detailing transactions where they disagree.
AC_153675905_1
10
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
TS] surs367P 5.38 Fallures or interruptions in service ‘Same type of incident as previous KEL at row 14 - same analysis. No impact.
with the harvesting process can
cause duplicate payment This KEL additionally states: as no customer accounts have been debited
transactions to be processed twice no further reconciliation is needed
By way of further explanation, the reason for there being 2 KELs is that this
issue occurred in both December 2004 and again in March 2009. However,
in both cases both Streamline and Fujitsu picked up the problem,
76 jharr832S Recoverable 541 ‘Acknowledgment that the recovery __I This KEL relates toa case where the system was behaving as designed, but I There would be no impact if the user
Transactions process (in relation to recoverable the user in branch failed to answer the questions correctly and did not follow I followed the recovery _ process
transactions) is a complex area. the recovery process properly. presented by Horizon. If the user
failed to do this there could be an
impact due to the user's error.
T7I_ carde464Q Failed 342 Particular difficulty in processing a__I This KEL does not indicate any software error. The specific scenario ‘As Mr Coyne acknowledges, in this
Recoveries recoverable transactions. in this described by this KEL would always appear as a failed transaction at a case there was no impact on the
instance the settlement of the
transaction had not been written into
the Branch database so the recovery
failure would have had no impact on
branch or customer accounts.
Branch, so itis highly unlikely that any cash was exchanged and the branch
or customer was out of pocket
All failed recoveries are monitored by Fujitsu and result in the exact
circumstances being checked out and a transaction correction issued in the
event of a discrepancy.
All failed recoveries are automatically identified in a daily report that the
security operations team receives (formerly MSU). They review this and raise
Peaks for any that require further investigation by the SSC team.
branch account.
AC_153675905_1
11
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
8] seng2037L
and acha959T
5.43
These KELs describe how various
transaction states may also indicate a
failed recovery.
These are both generic KELs describing how support should process
specific scenarios found on reconciliation reports. These reports are
generated daily for banking/debit card/e top up transactions where the
counters, agent and financial information (Fl) feeds differ in some way (for
example, the counter timed out a transaction but the Fl authorised it and ring
fenced the funds). They are sent daily to the security operations team. Each
transaction is checked and sometimes further investigation is required from
the SSC team via a Peak
These KELs describe legitimate states that can occur during reconciliation
following a failed transaction which is usually recovered from correctly. There
are 23 legitimate (normal process) states and 39 error states that a
transaction can enter, Each state has a specific meaning depending upon
the responses from the Counter, TPS and Fl. A legitimate state would be
that the TPS and Counter both know about a transaction but the Fi may never
send information about it (state 6); this is a legitimate state but it would
warrant investigation.
There would be no impact if the user
followed the recovery process.
presented by Horizon,
failed to do this there could be an
If the user
impact due to the user's error.
AC_153675905_1
12
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
19] dsed4733R 344 Example where transaction recovery I This KEL refers to a failed recovery report (report of failed recoveries) and inI This was a zero value transaction so
(no 20) has failed due toa wrongly named —_I particular some unexpected items in it there was no impact on branch
recovery script. Jason Coyne refers accounts.
toa Horizon system error arising The existence of the failed recovery report is evidence of routine robustness
because of incorrect Reference Data. I countermeasures in place to deal with failed recoveries. In this case, the If the transactions had a financial value
unexpected behaviour seems to have arisen from faulty Reference Data, then the issue would have had an
which was corrected within a couple of days. effect on the branch. However, it would
have been raised on Fujitsu's failed
This KEL refers to problems specifically with an ADCSoript-HPBB_REC1 recovery reports, investigated and
recovery script. HPBB stands for Home Phone Broadband and it appears reported to the POL via a BIMs report
that the transaction would have been a customer application for a Post Office
Home Phone Broadband service where information is collected from the
customer but no payment is made.
20] PSteed2847N Incorrect 554 Software issues resulting in Horizon I A user had Remmed In some cash to the wrong stock unit. When they Temporary financial impact which was,
Mathematical applying the wrong mathematical sign I realised this they carried out a reversal of the Rem In but, due toa software I obvious to the Postmaster (who
Sign when reversing transactions error the value of the Rem In was doubled instead. The reversal receipt would I reported the issue) and corrected by
(ie. a plus (+) instead of a minus (-))
have informed the user that something had gone wrong and in this case they
stated that their Rem In amount had doubled when reporting the issue.
Presumably they looked at a remittance report or balance report to see this,
‘A Rem In reversal is not a particularly common transaction and it was
prohibited as part of Impact changes in 2004. The user has attempted it for a
specific reason so if after performing the reversal it hasn't had the desired
effect you would expect the user to clearly notice this and raise a call with the
helpdesk to query it. Upon confirming the error the NBSC could then issue
an ‘error notice’ to correct any anomaly.
It took just 13 days from reporting to active a fast track fix
Post Office issuing an error notice.
AC_153675905_1
13
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
21) cardcS756N
Pouch Rem Out
Reversal
5.52
This is an example where the system
failed to reverse all items in a multi-
line pouch, meaning only the first
item was reversed.
This concerned the reversal of a Rem Out of a pouch, a rarely used process,
by which when a banch has prepared a pouch returning cash to the cash
centre and then realised that they either made a mistake (as happened in this
case) or need the money after all. It was not possible to rule out the
possibility that this was caused by a software issue, but it was not possible
to replicate it so this could not be investigated further.
The Rem Out reversal appears to have gone wrong in this case and only part
of the pouch contents was reversed, thus leaving some of the value still in
suspense.
Temporary financial impact which
would be picked up by Post Office's
cash centre reconciliation process and
corrected by a transaction correction.
22I GCSimpsont
O49L
Foreign
Currency
Discrepancies
5.54
All currencies in branch doubled up
following successful balancing eight
days previously.
The discrepancy (between the Horizon record and physical cash) was picked
up in branch and a call raised, It appears that this particular incident was
resolved by the branch upon monthly balancing
Due to the delay in providing the information to the development team, the
lack of any record of this incident having happened previously and there
being no further reports of similar problems, we are unable to confirm the root
cause,
Temporary financial impact which was,
resolved when the branch carried out
the next monthly balance (when it
declared the actual value of currencies
held in branch)
AC_153675905_1
14
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
23) MHarvey35271
Insufficient
Diagnostic Data
5.55,
Evidences that in certain
investigations there was insufficient
diagnostic data to be able to fully
diagnose an issue.
The KEL says:
‘As this is, at the moment a one off event and clearly no further progress can
be made at this stage, I have therefore closed PC113202 as "insufficient
evidence". However, any further occurrences should be sent to APS
Counter Dev for investigation.’
This is an issue in the transferring of copies of transactional data from
Horizon to Post Office's back-end systems. The specific data mentioned
here is not financial. The underlying issue was that there was unnecessary
validation of copies of Reference data being passed back to Post Office and
this validation was removed as part of Impact (a joint working body to
introduce improvements to the system and processes) in 2005.
No impact.
24) CObeng1123
Q
Stock Gains
5.56
Unexplained discrepancies (gains)
for different stock unit types (Cash
and Stamps) was reported. The
incident remained unexplained and
no record of advice having been
provided to Postmasters.
This was a complicated memory loss issue in branch. Extensive searches
were made for memory loss issues in test at the time and only one was found
and explained (which did not relate to this issue). It therefore appears that
this was a one off incident. Due to the passing of time, we're unable to
identify the cause, but if the counter software was running short of memory
we would expect the counter to display a warning to the user which would
have been seen.
Not known due to the age of the
matter.
25] DRowe1625K
5.57
This KEL is not available.
AC_153675905_1
15
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
26] _ dsed525Q PIN pad 568 There was a failure in the This was a faulty PIN pad. It prevented the Postmaster from carrying out No impact.
Failures Postmasters being able to transact I some transactions but this would not affect branch accounts.
various types of transactions
including payment transactions using
a PIN pad. An error message and
code were generated, and a new PIN
pad was the recommended solution.
27] surs30a1P PIN pad 568 There was a failure in the This appears to have been an issue with a corrupt customer card. Itwas set_I No impact.
Failures Postmasters being able to transact I up with no CVM (CVM is something on a card that indicates whether a PIN
various types of transactions of a Signature is to be used to authorise a transaction). SSC suggest that
including payment transactions using I this was an attempt to do a Balance Enquiry on a Credit Card (which isn't
a PIN pad. An error message and allowed). Reference Data should have prevented that being attempted and
code were generated, anda new PIN I it's not clear why it didn't. It could have been a corrupt card that the customer
pad was the recommended solution. I had
28) BrailsfordS22 PIN pad 568 There was a failure in the This was a faulty PIN pad. It prevented the Postmaster from carrying out No impact.
39K Failures Postmasters being able to transact I some transactions but this would not affect branch accounts.
various types of transactions
including payment transactions using
a PIN pad. An error message and
code were generated, and a new PIN.
pad was the recommended solution.
AC_153675905_1
16
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
29] carde219R PIN pad 5.69, 5.185 I This KEL indicated that PIN pad The KEL relates to a specific (one-off) hardware issue with the old Hypercom I No impact.
Failures related issues would usually result in PIN pad where the transaction was authorised but not confirmed due to the
the recommendation of anew PIN I hardware fault. The process is request, authorisation and confirmation,
pad, regardless of the error. In this I however, the final stage of confirmation did not take place. As per the KEL,
case a transaction had been declined I this was identified as part of the reconciliation process (i.e. the same DRS.
by the PIN pad but did not get reconciliation process referred to above) and passed back to Post Office and
reversed a Transaction Correction issued automatically. There was no impact on
branch accounts, however, the customer was charged twice and they should
have been reimbursed by Post Office's back end processes.
0] dsed4733R 5.92 Identifies multiple falled recoveries __I See analysis at line 19 above. See analysis at line 19 above.
occurring because of a wrongly
named recovery script.
2
AC_153675905_1 17
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
34] obengos933K 5.93 There was a loss of communications I This is further evidence of the failed recovery report doing its job and alerting I This may have caused a temporary
following network banking Fujitsu to failed recoveries to enable them to investigate them and make any I financial discrepancy
transactions and the printing of the I necessary corrections to accounts by sending a BIMs to Post Office. If this
customer receipts resulting in a had not happened, a transaction correction would have been issued as a
message to the data centre timing result of Post Office's own reconciliation processes.
out. Consequently, the Postmaster
was asked to follow recovery but the I The incident was caused by a complex ‘grey’ communications failure (i.e. the
transaction was only able to recover I network kept switching between good and bad; not solid good or solid bad),
partially. which the development team could not reproduce.
The KEL gives no reason to suppose that, even if this condition had
persisted, the backstop of reconciliation and Transaction Corrections would
not have corrected any resulting errors in accounts.
As per KEL, the failed recovery will be centrally reported and investigated via
the DRS reconciliation process.
32] wrightm33145 [~ Payments 5.116, 5.118 I Non-zero trading position, This is the same issue as line 1 above, See analysis at item 1 above See analysis at item 1 above.
J Mismatch
33] AmoldA2153P I Withdrawn BAIT This generates the same Receipts __I This issue was found during internal Horizon Online testing and fixed before I There may have been an issue if the
Stock and Payments Mismatch error Horizon Online went lve. branch had continued to rollover before
message (to KELs in lines 1 and 31) the products were reinstated, however,
but in fact relates to a mismatch In 2016, some products were withdrawn during a trading period when they _I in the 2 cases recorded, the branches
during the balancing of a stock unit I were still being traded. This is contrary to Reference Data procedures and _I were advised to delay the rollover until
that contains withdrawn product. caused an issue in branches that had traded those products. the products were re-instated so no
impact on branch accounts.
A fix was put in place involving the products being re-instated and the
branches affected rolled over successfully.
AC_153675905_1 18
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
34]_ballan§i7590 Payment 5118 Details 3 conditions that may cause a I This is a generic KEL ensuring that the event monitoring team raise a call WA (this is a generic KEL).
Mismatch receipts/ payments mismatch that _I every time a Receipts and Payment mismatch event is seen. It references
may impact on branch accounts other KELs that are known issues for specific cases. This is an example of
how our monitoring is made to work effectively. All of these calls get
investigated and may need manual correction to avoid errors in branch
accounts. From a Horizon perspective, none of the calls raised should
therefore be left without investigation / resolution.
35] _acha1357Q Incorrect 5.120 itis possible for discrepancies to ‘The issue related to declarations from the same trading period a year ago There may have been some financial
Declarations have been accepted by the becoming visible again and thus causing confusion. It only affected branches I impact, but there were reports
Postmaster based upon incorrect that had done a Stock Declaration the previous year but normally didn’tdo I available in branch that could have
declarations. The problem could arise I them. The fix was to change the archiving strategy so that all declarations been used to identify incorrect
due to old stock declarations not
being automatically removed from the
system. These
could only be removed by making
zero-value declarations or deleting
the stock unit then waiting overnight
before balancing.
that had not been updated for 6 months were removed. A check was made at
the time for any branches that had old discrepancies that might become
current again in the next 2 months (to allow for the archiving fix to be made)
and these were removed manually by MSC. A
The initial call was raised on 11th Feb 2011. A central workaround to avoid
further issues was implemented under MSC and the official fix was released
in June 2011.
declared amounts.
Further, a corresponding gain / loss
would occur in a subsequent trading
period so that would resolve the issue.
AC_153675905_1
19
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
36] _acha3145Q 5.121 Provision of a full support solution for ] This KEL is cited in the previous KEL dealt with at line 35 and relates toa There may have been some financial
incorrect stock declarations and stock balancing problem caused by the user doing some uncommon impact is is branch process at this
discrepancies. sequence — i.e. not caused by withdrawn products. stage, it is effectively as if the SPM
had wrongly declared the cash or
stock and the system will warn them
that this does not match the calculated
value. In this case it was an old
declaration that got _included.
Redeclaring the cash/stock will fix the
issue. If they do not they will roll with a
loss this BP/TP but have a gain next
TPIBP.
37] _allend1645p Horizon’ 5.129 Provides an example of Horizon’s Horizon allowed the clerk to select ‘Debit card’ as a method of payment and _I This is a case of the Postmaster being
Interface weak interface controls and lack of I_I later switch to ‘fast cash' at the end of the customer session. This was a responsible for errors made by staff.
Controls data entry validation. In a single sales I subsequent user error which involved the user at the branch failing to take I This would have shown as a
transaction the user was able to payment of £500 for the 540 euros. discrepancy caused by user error in
select and enter different methods of not taking the payment that was due
payment (Debit Card and Fast Cash), from the customer and the Postmaster
Horizon allowed the transaction to be would be liable for this,
settled via Fast Cash when the Debit
Card payment method had already
been selected,
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
2
8} acha621P.
5.130
The correct screen to successfully
process a cash pouch did not appear
resulting in the clerk in an outreach
branch inadvertently doubling up the
amount of cash recorded, The issue
appears to have been caused
because of an earlier system logout
or inactivity which in turn resulted in
incomplete checks being conducted
by Horizon post logon.
This issue, also referred to as the ‘Dalmellington’ issue is dealt with
substantively in the second witness statement of Torstein Olav Godeseth.
Temporary financial impact which was
rectified by a Transaction Correction
being issued
g
9} EJohnson393
7R
Horizon
Interface
Controls
5.132
This enabled Postmasters to carry
‘out “Rem In” transactions without a
value being entered.
Whilst Remming In currency it is possible to create a transaction with a
positive quantity and a zero value
This issue was caused by the Reference Data test team (not following the
correct process). The functionality was changed the following year so that
amounts did not need to be entered.
Auto Rems (introduced around 2004) meant that the content of cash and
currency pouches was sent to the branch electronically so when a pouch was
delivered, the system automatically told the Subpostmaster what the content
was and used that value for the Rem in rather than asking the Subpostmaster
to key it in.
No impact.
AC_153675905_1
21
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
40] PSteed145J
Phantom Sales
5.133
Involved ‘phantom’ sales appearing
‘on the Horizon counter screen but
which had not been selected by the
user.
These “phantom” sales were caused by hardware problems and fixed by
replacing hardware.
No impact If the transaction related to
stock, when the branch declared their
stock and cash the discrepancy would
cancel out (e.g. a sale of stamps would
reduce the stock of stamps and
increase the cash figure by a
corresponding
balancing
stamps should be declared and this
out the effect of the
amount;
the correct number of
when
will cancel
phantom sale of stamps).
41] pearrolli235R
‘Screen Freezes
5.133,
Instructions on how to deal with
environmental issues and hardware
are contained in this KEL. Jason
Coyne says "it is not known how
widely these were distributed to
‘SPMs", implying that they should
have been.
These instructions (which relate to issues that do not concern bugs or errors
in Horizon) were distributed to those who called for help via this KEL.
No impact on branch accounts
AC_153675905_1
22
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
42] jharr1323L 5.136 Example of a successfully recorded I This relates to a fishing rod licence request not sent to the Environment No impact.
transaction initiated in a Post Office I Agency and was only detected at a later date. In this particular instance, the
branch (where a customer receipt transaction had been reversed by a user; this is not a software issue, Once a
was generated) which failed to transaction is reversed, all relevant data is discarded and not sent to the AP
appear in the Post Office Data Client as its effectively as though the transaction never took place (Fujitsu
Gateway. only keep Post Office Data Gateway records readily available for 30 days) but
it is committed to the audit store and therefore any additional investigation
would have needed to be undertaken by audit)
Itis not known why the reversal took place. This could be due to fraudulent
activity or could be that the customer sought a refund and the money was
refunded.
43] MArris34331 3137 The ability of Horizon to erroneously __I This was a software bug which allowed a transaction fo be recorded twice No impact (fixed in testing).
(no 46)
record the same transaction twice
after a session transfer to a different
counter. This happened with both
NS&l (National Savings &
Investments) and Network Banking
(NWB) transactions. The KEL was
passed to a development team to
provide a bug fix as part of the S60
rollout but it is unknown if this was
ever resolved
after a session transfer and was a fairly rare circumstance. This issue was
picked up during internal testing and despite numerous attempts
development were unable to recreate the scenario. There are no Peaks
which refer to this KEL.
AC_153675905_1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
44) CharltonJ2752
T
Previous Key
Software Error
5.139
Counter level corrections made via
the “Previous” key led to both the old
value and amended value being
stored and used in error in the
transaction. A fix was released in the
Live environment eight days after the
issue was first raised
A software error in a PDL file meant that when a user used the “Previous”
key for a transaction that used an ADC Script, old and amended values were
‘stored and used. This resulted in an incorrect transaction for the sum amount
of both the old and amended values being added to the sales basket.
As the user has used the "Previous" key to go back and amend a value, it
should have been obvious to the user if then a completely different value item
is added to the sales basket. If, for whatever reason, this was not noticed
then the the customer will end up being overcharged as the system will ask
the user to take a larger payment. Assuming the user does what the system
says there will be no impact to the branch accounts,
This issue affected 3 products and only occurred when "Previous" key was
used to correct the amount entered, It was fixed 8 days after it was spotted.
A search was made for all branches that had used those products twice
within a session and the results were sent to Post Office
No impact.
AC_153675905_1
24
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
45) = SSur343P.
5.141
Example of a declined network
banking
transaction that resulted in money
being taken from the customer's
account.
‘An error in network banking caused the customer's account to be debited
although the transaction failed at the branch and the Postmaster was told at
the counter that the transaction failed. Banking transactions with a response
code of 26 (FI Unavailable, Try again later) would be recorded as zero value
transactions at the branch, a DECLINED receipt would have been produced
and so no money should have been handed over to the customer. Therefore
no impact on branch accounts. On rare occasions the financial institution
may have debited or credited the customer bank account despite us not
receiving the authorisation. in these instances, if the automatic reversals fail
to resolve matters then the issue would be picked up as part of the DRS
reconciliation process meaning Fujitsu would inform Post Office of what has
happened at the counter so that they can liaise with the financial institution to
ensure their systems match so customer is not out of pocket. The root
cause was between the NBE and the Fl, which is outside of Horizon.
No impact.
46] LKiang3526R
5.142
Examples of E-Pay transactions
crediting the customer account twice
although only one payment has been
taken.
This was caused by two authorisation agents being active at once, when only
one should have been, resulting in the phone being credited with £10 twice
and ePay charging Post Office or the phone provider (depending on the
arrangement between the two) being charged twice, meaning the customer
would have got two sets of top up for the price of one.
This is a back end system problem which would be picked up by counter
measures, causing a BIMs to be raised.
No impact.
AC_153675905_1
25
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
47] SSurs310P 5.142 Examples of &-Pay transactions Similar to the KEL above at line 46. No impact.
crediting the customer account twice I This is a back end system problem which would have had no impact on the
although only one payment has been I branch accounts. Despite a phone being topped up twice at the branch only a
taken. single top up would have been recorded in branch along with the required
payment for it
Again, this is a back end system problem and would need to be resolved by
Post Office centrally and would not impact branch accounts.
48] pothapragada 5.165 Jason Coyne says "itis The issue was the ability to declare stock that could not then be transacted _I This may have had a financial impact
04359R
acknowledged that simple fixes
ought and were implemented to
either fix bugs or provide additional
data validation checks"
(due to Reference Data rules),
The only impact on a branch's account were if the branch were to actually
declare that it held an item of such stock. This is unlikely as the item had
been withdrawn and should be returned to the stock centre. It should also be
noted that most branches do not undertake stock declarations as stock is
normally manged using stock adjustments (which didn't have this issue)
Should the stock be declared by mistake by user-error, then a further
declaration of the correct (i.e. zero) holdings would resolve the issue.
but if so it would be due to human error
(ie. declaring that it held an item of
stock that it couldn't transact). This
discrepancy would be removed if the
branch accurately declared that it had
no such stock,
AC_153675905_1
26
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
49] Marris4123N 5.165 Jason Coyne says “iis This was a problem observed during test in Disaster Recovery for DVLA No impact.
acknowledged that simple fixes transactions - a very rare circumstance, which should be handled correctly,
‘ought and were implemented to but would have no impact on branch accounts in the unlikely case that a
either fix bugs or provide additional _I branch encountered the issue
data validation checks"
When recovering a failed counter, the user is asked to input data from a
receipt. To handle poor typing there is a check sum which should ensure that
it has not been altered. This bug relates to the fact that not all the data
entered is controlled by the check sum. Therefore, when a tester deliberately
input incorrect data, the system did not detect it. NB this did not include
financial data
50] acha2230K ‘Additional 5.186 Jason Coyne highlights a problem I This issue, also referred to as the ‘Local Suspense Account issue is dealt I This issue had a financial impact
Checks with additional checks which were _I with substantively in the second witness statement of Torstein Olav which was resolved by Post Office
implemented to identify system
errors/ inconsistencies when
balancing
during branch and references a note
within the KEL as follows
"This should never happen -
something has gone horribly wrong.
Or possibly the checks haven't been
implemented as intended."
Godeseth.
writing off discrepancies.
AC_153675905_1
27
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Si] dsed20498 5.187 Jason Coyne suggests that this KEL I This KEL relates to withdrawn products that were converted to cash on No impact if the correct process is
highlights the lack of system rollover but the losses are carried forward into the next period instead of followed for returning the withdrawn
communication and/or support being dealt with there and then products to Post Office. In the event
communication in respect of certain that the process is not properly
system features which could Withdrawn products should be sent back by the Postmaster to Post Office _I followed, a Transaction Correction can
subsequently result in errors. so the branch is not holding stock that cannot be sold. be issued to correct any impact on
branch accounts,
If this process is not followed, the branch will be left with a loss at the next
trading period and could be corrected by a Transaction Correction. The stock
will also be converted to cash if the Postmaster has purchased it personally,
for example. The fix was made to make this clear to the Postmasters.
Jason Coyne implies this was a bug which took 6 months to fix. There was a
minor bug in that the cash value of the withdrawn stock was not included in
the current rollover, but delayed until the next rollover. However there was no
specific loss to the branch (other than the value of the withdrawn stock which
they were responsible for as described)
52] _acha3250R 5.189 The reconciliation process used by I This was an issue with Post Office's back end reports, caused by timing No impact.
Post Office to assist with identifying I issues (APS transactions arriving a day late) which were outside of Post
any accounting differences is not Office or Fujitsu's control.
able to easily identify genuine
differences and/or differences This caused discrepancies in certain back-end reports, which could however
resulting from external APS be understood by cross checking other reports. This is a “false error’ being
transactions from old trading dates. _ I reported by reconciliation relating to transactions occurring around the end of
day, being 7pm.
There is nothing wrong in the actual transactions — just an error in the way
that reconciliation totals are calculated,
AC_153675905_1
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
$3] acha1941L
76 During the recovery process (when
‘some transactions recover but others
fail to recover) itis only the recovered
transactions printed on the receipt.
The disconnected session receipt
should also identify those
transactions not recovered. These
are printed for the Postmaster to
retain.
The KEL says:
“This is not really a problem, itis just confusing when investigating a state 4
call. The Disconnected Session receipts will show all the transactions in the
session. The successfully recovered transaction needs no reconciliation."
This shows that any incomplete information went not to the branch, but to
someone in Post Office or Fujitsu investigating a state 4 call
As there is no reconciliation needed,
there is no impact on branch accounts
54! surs1147Q
Failed Recovery
77 This relates to the solution to a failed
recovery requesting the user to log
onto the relevant counter and start
the recovery process, but leave the
counter displaying the system error
message.
While the advice to the user in the KEL is somewhat counter-intuitive and
indicates that nothing should be done and the user should wait for the
counter to time out, leaving the system error, there is nothing to suggest the
recovery was not resolved. The root cause of this issue was an error in Post
Office's script relating to the Dangerous Goods products, that resulted in
recovery failing. We're unable to tell whether or not the script was corrected
This would have no direct impact on branch accounts, but clearly would be
very inconvenient as the counter is out of action.
No impact.
55] wrightm33145j I __ Payment 79,741 I Cited by Coyne. This issue, also referred to as the Payment Mismatch issue Is dealt with See item 1 above.
Mismatch substantively in the second witness statement of Torstein Olav Godeseth.
56] boismaisons1 Disc Space 9.12 This KEL describes the running This is an issue with hard disks. Disc space sizes have nothing to do with No impact.
328M Sizes commands on counters to assess __I branch transaction data which in any case at the time (2012) was not stored
disk space sizes,
in the branch.
AC_153675905_1
29
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
37] SeemungalG5 I Transaction 9.49 Records an instance where This was an issue when posting transactions from old Horizon to BRDB prior I No impact.
19a Amendments transaction toa branch migrating from old horizon to Horizon Online. In some cases,
amendments carried out are causing I such transactions fell foul of validation in TPS and needed to be amended
exceptions, before being sent to Post Office's backend systems. Such amended
transactions were also posted to BRDB. Any amendments were related to
the trading period they related to and not any financial values. They were only
in BRDB to be used if the branch migrated during the current trading period
from Old Horizon to Horizon Online.
Tip Repair is a back end process to make sure all required transactions get
sent to the relevant external systems. There is no effect on the branch
accounts
58 MHarvey2255 I Corrective 9.50 Records the manual addition of This is a back-end balancing issue caused by missing or invalid Reference _I No impact.
P (no 64) Balancing corrective Data. The insertion of correction records is done into the TPS database to
Transactions balancing transactions inserted by _I allow the branch data to be forwarded to POLSAP. There is no effect on the
SSC affecting the TPS system. Branch Database therefore no effect on the Branch Accounts. It is not
understood why this is marked as remote access as the work is done within
the Data Centre
AC_153675905_1
30
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
APPENDIX 2
KEL Fujitsu's analysis
ID Nickname Summary of the KEL Analysis Impact on branch accounts
achad23K Cash Button I The Cash button on the settlement screen I This is not a bug, rather itis a feature of how the system operates. I No impact unless the system is misused by
can be used for either a receipt in or a branch staff.
payment out of cash. Horizon decides on
the context depending on whether the
stack total button says TAKE or PAY,
‘acha488S Large This involved both transactions being I Although the clerk took the money from the customer, the session I No impact
Transactions I completed in a single session. wasn't settled because £1m is too large for FastCash. The
settlement should instead be entered as two £500,000 cash
payments. This is an issue with hitting system limits with very large
transactions over £1M. This would be very noticeable and there is
an avoidance action to take two smaller payments.
Looking at the Peak, the Transactions were actually recovered
automatically by the system (AP Txns were recoverable in old
Horizon) so no impact on branch accounts once recovery was
carried out.
‘achaS08S Postmaster reports problems with I This was a bug in the handling of multiple bags of coins when I Temporary impact that would have been
remming out, in particular differences I being Remmed Out. It was fixed by a code fix issued in April 2007 I identifiable from the reports available to
between the two receipts which are printed I and fully rolled out by June 2007. Subpostmasters, The KEL description states that
after the pouch barcode is scanned. “differences between the two receipts which are
AC_153675905_1
3
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The original problem was found on 12° Feb 2007 and was I printed after the pouch barcode is scanned...
presumably due to a software update rolled out at that time. I The second receipt (Office Copy) only shows one
Investigations were carried out and a list of affected branches was I bag of each”.
generated (this is no longer available) and provided to Post Office.
NBSC was informed about a workaround to the issue. It would have been rectified by a transaction
correction.
achaS22T Cash This involved two separate cash I This is not a bug, rather the incident appears to be caused by I Impact caused by human error.
Withdrawal I transactions most likely by two separate I human error of the user not reading the screen carefully when
customers as follows: doing a withdraw limit CAPO transaction after failing to settle an
earlier customer basket.
* customer 1: given £200 in cash
* customer 2: given £320.90 in cash
(being the total value of the stack)
The loss was caused by the user not
setting stack in between banking
transactions
‘acha2140S__ I £m Cheques I If the branch holds cheques to a value of I This will be rare and if encountered it will be highly visible by the I No impact.
more than £1M, then the value of cheques I Postmaster.
cannot be adjusted using the normal Stock
Adjustment mechanisms. The best thing to do would be to rem out the Cheque for £1M and
then any further adjustments can be handled as normal
acha3347Q Ifa stock unit carried forward from Horizon I This was an issue for a branch following migration from old I No impact
is deleted before the first trading period Horizon to Horizon Online and only affected a branch if it had
rollover on Horizon Online, the check for I deleted a Stock Unit after migration and before the first rollover.
AC_153675905_1
32
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
‘jast stock unit’ may not be applied
properly, and all the stock units can roll
over without Local Suspense being
cleared.
Local Suspense for a particular trading
period has to be zero before the branch
can be rolled over. Once a stock unit is in
the new trading period, it can put
gains/losses into Local Suspense and
clear them, but this has no effect on Local
‘Suspense for the old trading period - think
of them as two separate containers.
The issue would be clearly visible as the Branch would be unable
to rollover without following a complex work around.
acha3610P
‘Advantageous
Exchange
Rates
There are advantageous exchange rates
for transactions over certain limits (shown
on Foreign Currency report as DDE for
Euros, DDU for US dollars). In this case
the limit was £500.
The user pressed ‘Buy Euros’ and entered
600. The exchange rate shown was the
standard exchange rate, not the rate for
transactions over the 500 limit, the reason
being the £500 limit had not been reached.
This appears to be a misunderstanding of what exchange rates to
use, The “over 500" rate applies to when the sterling value is over
£500 (not the Euro value),
No impact.
‘acha4221Q
The clerk went into the ‘Delivery’ menu and
scanned two pouches (one of currency
and one of coins). The second pouch was
recorded twice on the system, resulting in
This was a bug in the early days of Horizon Online following an
unusual (but valid) sequence of events. It was fixed on 19" April
2010.
Financial impact would have been clear to the
branch because a duplicate receipt was printed
It would have also been identifiable from the
reports available from Horizon
AC_153675905_1
33
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
a loss of £80.
Two Remittance In slips relating to the
second pouch were output, both identical,
as well as one for the first barcode.
It was resolved by a transaction correction being
issued.
achad353P The counter froze while cashing a Postal I This was due to an invalid Postal Order with a negative value being I No impact
Order and now recovery won't complete so I set up by an external system which resulted in a counter being
the counter could not be used The counter I frozen.
was rebooted, but when they logged back I A fix was applied to check the value of Postal Orders coming from
in they got a Postal Order encashment I external systems as being positive and this was applied on
transaction recovery message for a I 15/8/2011. A system fix was made to allow recovery to be
negative amount, followed by the Invalid I bypassed on this counter to make it useable again.
Value message again.
0. achaS226J When a branch puts through a bureau I This issue was fixed in October 2010 No impact.
transaction in excess of £5,000 a message
should appear on the screen to remind the
user to ask the customer to take two forms
of ID from the customer to conform with
Anti Money Laundering regulations. This
reminder prompt is not appearing on the
Horizon Online counters.
1 acha5259Q
The Postmaster wrote a discrepancy of
£167.17 to Local Suspense, then this was
cleared from Local Suspense as normal
and the Postmaster selected to make good
the losses. At this point the system printed
out a final balance report for the trading
period with the cash figure amended and
It appeared to only affect branches balancing in April 2010 and 33
branches were identified as being impacted. Details of these
branches were passed to Post Office.
This was an issue found in the early days of Horizon Online and
was resolved in July 2010.
‘Temporary financial impact which would have
been cancelled out in the following period by a
corresponding discrepancy.
AC_153675905_1
34
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
nil discrepancy. Normally the system
would then come up with a message to
confirm rollover but instead went back to
the screen asking how the discrepancy
was going to be made good,
2. I AChambers1833
Report Issues
A transaction for £6.67 was done at 17:23
and was settled at 17:25. The daily APS
transaction report was done on a different
counter at 17:25 and did not include the
APS transaction.
‘timing issue to do with printing reports at a counter. There was
no impact on the branch accounts — just confusion due to a
transaction being missed from a report and the report not being re-
printed when it appeared,
No impact.
3. I AChambers2252
R
Postmaster sold some foreign currency
(1000 euros, £750). The
Postmaster realised the transaction had
sale value
been settled to the wrong product in this
case being cash instead of debit card.
Existing Reversal was used to reverse the
transaction, and then re-run correctly.
When it came to balance at the end of the
trading period, the currency stock holding
on the system was too high by 1000
Euros. When corrected, this gave a gain
on currency of £720 and a cash loss of
£750, being a net loss to the branch of £30
This was not a bug, rather an issue in how a Currency transaction
was incorrectly reversed on old Horizon.
Foreign currency transactions consist of two parts: the currency
itself, which has a value based on the exchange rate, and margin,
which is added to cover the cost of the transaction. When the
transaction was reversed, the Postmaster entered the transaction
for the cash settlement part of the transaction. While the
Postmaster believed the whole transaction had been reversed, it
had not as the margin had not been reversed. When the stock unit
was balanced, the wrong number of Euros became apparent. The
stock holding was corrected by the declaration of the actual
number held. Again, this did not correct the margin, which is
generated as part of the currency sale and is not directly linked to
the stock
The way Reversals are handled on Horizon Online means that this
Impact, but guidance on how to correctly perform
an existing reversal was all that was needed to
rectify it.
AC_153675905_1
35
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Sort of issue can no longer happen.
4. I AChambers4134
R
Multiple quantity for stamps/postage label
affects cash settlement or subsequent
transactions
When a quantity greater than 1 is entered for a Smartpost
transaction, the Quantity is not reset to 1 when the user moves on
to the settlement screen.
If the transaction is settled to Fast Cash / Fast Cheque or Debit
Card, this doesn't matter, but some users habitually use the Cash
(F2) button to enter the cash presented by the customer, then give
the customer change as indicated by the new stack total. If this is
done, the cash amount entered is multiplied by the Quantity and
hence the new stack total is wrong
This was fixed in December 2005 (Reference Data fixed).
This may have had an impact but a user should
have been able to spot it and the sums involved
are likely to be small due to the issue affecting
mails products.
5. I AChambers4413
Q
Receipts and
Payments
Mismatch
This appears to have been an issue with
doing a Transfer Out which was not picked
up correctly when balancing.
This was picked up by Reconciliation Reports(looking for receipts
and payments mismatches) and investigated.
Temporary financial impact which would have
been cancelled out in the following period by a
corresponding discrepancy
6. I AChambersS711 Postmaster balancing on counter 1, then I This occurred due to a counter failure around the time EOD I No impact.
K completing on counter 2 due to counter 1 I activities were happening and thus resulted in a mismatch in
timing out, causing a discrepancy between I Reconciliation reports
reports.
7. Agnihotriv245L Horizon Online I The last digit of the exchange rate is I No Financial impact. It was fixed in September 2010. No impact.
Exchange
Rate
occasionally being displayed incorrectly on
the rates board.
The system does use the correct values to.
calculate the rates -the issue here is with
the display only. However, because the
AC_153675905_1
36
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
display is on the Customer facing rates
board there is potential for annoying
Customers as they may get a slightly
different rate to that advertised on the
board,
8. I AOConnor1581
Transaction
Limits
Failed to Harvest a quantity of 11743997
pennies as being too large.
This is another problem with limits. The branch tried to declare a
cash holding of over £100,000 in 1p and this hit a limit in
Reference Data. The fix was to police the system limit on
declarations.
No impact.
9. I AOConnor52571
The Postmaster remmed in a cheque for
£3,200, however, it did not show up on his
balance snapshot or adjust stock, but itis
showing in his Suspense Account.
This is a user error in how Post Office cheques were handled
Following advice and Guidance the problem was resolved.
Temporary impact caused by user error.
0. arnolda229R
‘An Open Value Encashment for £5.00 was
performed, the transaction was authorised
and added to the Basket but the counter
crashed before the Basket could be
settled. On Login Recovery was invoked
and a ‘Recovery Failure’ receipt was
printed for £0.00.
There was a typographical error in the script causing the issue
This issue was found during testing of Horizon Online and fixed
before the first counter went live.
No impact (resolved before Horizon Online went
live).
1. I ArmoldA2341L
Currency Code
Validation
All currency codes, in terms of "IMoney”
objects should be ISO-compliant, or
verified by £-sign However, we do not
validate this at all anymore, in order to.
support future currency codes. This
means that junk codes are accepted.
This was an issue identified by developers regarding the way
currency was handled within the counter code. It had no impact in
the real world it was closed.
No impact
AC_153675905_1
37
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
2. I _ ballantjo20J
Postmaster states that she has sold a
Lyca top up for £10, the message appear
unable to connect to the data centre then
logs Postmasters out.
The transaction request has been
authorised and the reversal may not be
effective which will cause an 'E21"
reconciliation error.
There were 2 issues here:
1.There was an issue with how Reference Data was generated
which resulted in some counter scripts failing. This was fixed on
20/08/2010 (2 days after this problem occurred);
2. The Ref data issue caused an e-pay transaction to fail and the
Postmaster didn't handle the recovery correctly
The failure to handle recovery correctly may have resulted in a loss
of £10,
This may have resulted in a small impact due to a
failure to follow the correct procedure.
3. ballantj2547K
The Transaction Processing System Total
and Counter total values for the Number
and Absolute Quantity columns are the
same but the Absolute Value for Counter
Total is greater than the corresponding
Transaction Processing System Total by
£14.80.
If the session nets to zero (add up all the
SaleValues for the same Sessionid) no
reconciliation is needed. If it doesn't, a
correction must be made to send the data
to POLFS (see <a
href=kel_view_kel jsp?KELRef=MaxwellG
460L>MaxwellG460L</a>) and the PM
may need to be told about a possible
receipts and payments mismatch, or at
least watch out in case one is raised
This is a problem with Smart Post which seemed to write slightly
corrupt transactions in that there were missing attributes required
by back end systems and reconciliation systems, but are complete
as far as the branch accounting is concerned.
This therefore has no impact on branch accounts, but does result
in reconciliation errors which are fixed by amending the transaction
copies in the backend systems. The fixes are always to dates and
not to values.
This KEL refers to amount mismatches, but the amounts used by
the reconeiliation are different from those used by accounting.
They should be identical, but in this case are not. The accounting
values are the correct amounts.
No impact
AC_153675905_1
38
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
4. I ballantjs342L
Reconciliation picked up a scenario
associated with a failed banking
transaction. Specifically, the request from
the counter never reached the Branch
Access Layer.
The application event log shows AdminCfg
receiving data and causing a
VPNKeyChange
In this case the request never reached the
Authorising Agent and therefore no money
was requested, the CO did reach the
Authorising Agent but was unexpected and
has caused this reconciliation incident.
Instruct MSU that no reconciliation
required.
‘As the transaction failed there was no impact on the branch
accounts.
This did identify some issues in the way that failed transactions
were handled and why they resulted in reconciliation errors and
these were fixed in April 2010
No impact
5. I bambers3553L
symbol
Currency
Specification
A MoneyGram Send transaction was
initiated. There are 3 options to define the
amount being sent:
i. € including fee
ii. £not including fee
iii, I Receive Amount excluding fee.
The 3rd option allows a customer to
specify an amount to be sent in local
currency, such as $300, in which case the
receiver will get $300 and the send amount
This was an issue found during testing of Horizon Online and
relates to a “E” sign being displayed when the user is being asked
to input an amount in another currency to a MoneyGram
transaction.
It was agreed not to fix this until a later time (not clear if it ever was
fixed, however, the MoneyGram product was re-engineered in
2015 so it would have behaved differently after that time anyway).
This would have no impact on branch accounts, but may have
caused some mild confusion to the Subpostmaster.
No impact.
AC_153675905_1
39
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
is calculated in sterling. However, at the
input of the receive amount, the prompt
appears (correctly) as ‘Amount in USD’ but
the input box (which is a currency
datatype) has a '€’ symbol present. This
leads the user to think that they have
selected the wrong option and would lead
to incorrect amounts being entered here
Please see the attached screen shot for
evidence.
6. I bambers4236K
Electronic Top
Ups ("ETU")
ETU E-voucher for £10.00 is erroneously
declined as a New Reversal.
The basic problem is that, to support ETU
Reversals, we rely on the Authorising
Agent remembering details of the original
ETU transaction. In a CTO we only have
an Agent Simulator and it is not configured
to handle ETU Reversals. Given the
simplicity of the Simulator it would be very
difficult to support ETU Reversals.
This as a restriction for CTOs but is not
documented in DES/GEN/REP/0006 nor
REQ/CUS/STG/0004
This is an issue in the Counter Training service in that it doesn't
support reversals of E Top ups.
It was discovered during testing and was agreed that this would be
a restriction on the functionality supported for Counter Training
As this only impacted counter training then there is no impact on
any Live Branch accounts,
No impact.
7. BluerP5546R
Upside Down
Pound Sign
For PING transactions (Transaction
Acknowledgments) £ signs will appear as
upside down question marks in the training
This was a cosmetic issue when processing Transaction
Acknowledgments on a Training Counter in that the “E" sign was
displayed incorrectly. This has no Financial Impact as it was only
No impact
AC_153675905_1
40
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
counters.
affecting training counters.
This issue was spotted in internal testing and we had no reports of
this issue from Live Counter Training Offices. A fix went live in
January 2011
8. cardc235Q
Drop and Go
The user initiated a Drop and Go
transaction for £100 which failed due to
timeouts. Following the failure, a success
message was displayed. The user settled
the transaction and the customer handed
over £100, The customer checked the
balance and stated that the top up had not
gone through so the clerk performed
another Drop and Go transaction which
was successful. The customer has paid in
£100 but the branch account has been
debited by £200. Accenture verified that
only the second Drop and Go top up was
successful.
This was a problem in handling errors correctly in a Drop and Go
script provided by Post Office.
This was passed to Post Office to fix the scripts.
This would have caused a loss in the branch
accounts, although the issue was identified by
the Subpostmaster and it would have been
resolved by a transaction correction.
9. carde339P
Receipt
Printing
A Transfer Out of £517 cash was done on
counter 14, session id 14-2966714-1. A
corresponding Transfer In was done
immediately afterwards on counter 2,
session id 2-2046304-1. Before the
Transfer in had completed, a receipt with
the wrong session id (14-2966714-1) was
printed. After the Transfer In had
completed, the correct receipt was printed.
The system is behaving correctly and there would be no financial
impact on the branch account, but the Postmaster may be
confused as to what exactly has happened
No impact - the Transfer Reports and the
Transaction log will show exactly what has
actually happened.
AC_153675905_1
41
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The extra receipt, which is almost identical
to the actual Transfer In receipt, is quite
confusing. It may lead the Postmaster to
suspect that the transfer has been carried
out twice, when in fact it has not. Advise
the Postmaster to use the ‘Preview’ rather
than the Print button if they wish to view
the individual products being transferred.
0. I cardea27T The transaction appeared as a state E26 I This was an issue with failed banking transactions and then I No impact. Following a failed banking transaction
on the NB102 Section 2 Link report. recovery from a subsequent counter failure using the same I the next banking transaction may reuse the same
identifiers. Some of these appeared in reconciliation reports when I unique reference. This will cause that transaction
This issue has also resulted in a I they shouldn't be and so causing additional work to Support I to fail also as the authorisation software knows
transaction being reported as both a State I teams the transaction previously failed and will not pass
4 and State 6 - see PCPC0244934 it on to the Fi.”
However as these are all failed transactions they are all for zero
PC0197368 was fixed and released as I amounts and so have no impact on branch accounts.
part of release
CTRO1_22_01_00_RELEASE (June
2010).
Unfortunately the change was regressed
as part of a BAL change in February 2015 -
see peaks PCPC0243030 and
PCPC0241771. Another fix is in progress.
7 carde2326R The user will usually count out the cash to I The system is operating as designed and no change was I If the correct procedures were followed there
End-of-session
be paid before pressing Fast Cash,
because once they have pressed Fast
requested by Post Office
would be no impact.
AC_153675905_1
42
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
Sales prompts -
usability issues
Cash then the amount to be paid out is
reset to zero and the stack disappears.
However, if the transaction results in a
sales prompt then the stack is not cleared
and the amount payable remains on the
screen. If the user selects the sales
prompt and transacts another product, this,
is added to the stack and the ‘Total Due To
Customer’ is updated by the relevant
amount.
If the user is distracted or busy then they
potentially pay out the new amount in
addition to the original amount. The system
is working as designed and Postmasters
should be referred to the NBSC.
There may be confusion in relation to the way Sales Prompts are
handled at the end of a session.
2. carde3335R
Vodafone Text
Pack Vouchers
being declined
A call was raised with e-pay to determine
why requests were being declined. This is
e-pay's response: The reason for the
Vodafone £10, £15 and £20 Text Pack
vouchers being declined by e-pay is
because they were deactivated on request
by Vodafone. We sent out a Product
Configuration document in May detailing
this change. This document was sent to
Dave Cooke at Fujitsu as well as Clare
Tetley and lain Gilbert at Royal Mail, The
Not a bug. Certain E Top Up products had been withdrawn but
the Reference Data had not been updated to remove them from
the counter. This meant that e-pay declined the requests.
Following the investigation, then the Reference Data was updated
to remove the products from the counter the following weekend.
No impact.
AC_153675905_1
43
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
deactivation was rolled out on June 1st.
3. carde3415N
When a branch migrated from Horizon to
Horizon Online, differences were reported
between the 'Pre Quantity Move’ and ‘Post
Quantity Move' figures.
This was a problem in the migration of a branch from old Horizon
to Horizon Online.
The report produced pre-migration on the old Horizon didn't take
into account a Transaction Correction carried out in the last trading
period so adjust stock levels. The report produced post migration
was correct.
No impact - the figures post migration were
correct and the issue down to an inaccurate pre-
migration report.
4. cardc5946P
Halifax 7 Bank of Scotland bank cards
declined with response 05 - ‘do not
honour’
Branches with an “& in their name were resulting in Banking
transactions being declined by Halifax / Bank of Scotland.
No information was given by Halifax / Bank of Scotland regarding
why they are declining the cards. This was fixed by May 2011
No financial impact on branch accounts
Ss. I CCard1223Q Counter hangs when attempting to clear I It would appear that an Invalid option was presented on the menu I No impact
local suspense during stock unit rollover. I of options available when settling local suspense. This seems to
have been fixed shortly afterwards.
6. I CCardd650N The stock unit balance report only includes I Buttons were introduced fo record when cash was added and I No impact
figures for Add or Remove Cash I removed due to variances being spotted. However the behaviour of
transactions done in the current balancing I these was not carried forward from one balancing period to the
period. It should however show the I next and so caused confusion
cumulative total since the start of the
trading period When the problem was identified, the buttons were ‘padlocked”
and a fix was issued in March 2006.
7. I CharitonJ222L Log on event timestamp canbe after the I The log on event appears to be recorded at the time the log on is I No impact.
log off and other associated events when
looking at rep events for the HBS Kiosks
processed by the HBS server, but the other rep events are
recorded against the "DateTime" field in the incoming message.
AC_153675905_1
44
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
8. I CharltonJ2752T
‘See item 24 of Appendix 1
‘See item 24 of Appendix 1.
See item 24 of Appendix 1.
9. I CObengi123a
See item 44 of Appendix 1
See item 44 of Appendix 1.
See item 44 of Appendix 1.
was an unsettled banking transaction
requiring recovery, which had been done in
stock unit AA.
Stock unit AA was still in TP 12 BP 4,
Stock unit BB was already in TP1 BP 1
Recovery completed successfully,
correctly writing the transaction and its
settlement into stock unit AA, but for TP 1
BP 1.
0. I achad34oK Reconciliation reports relating to declined e-I This has no impact on the branch but caused unnecessary work _ I Zero value transactions have no financial impact
pay transactions are not clearing down for the Fujitsu and Post Office's reconciliation teams. Issue was__I on branch accounts.
correctly. The affected transactions are I fixed on 11/10/2010.
zero value, and have been declined by e-
pay.
7. I achad745R This was an issue relating to back end KEL suggests issue was with reconciliation reports not being Not known,
reconciliation where there was a £20 (manually) processed correctly. Peak PC0219762 applies. This is
difference between 2 totals relating to due to a recovery performed at the branch on the following day
millions of pound worth of LINK which reversed the transaction. IT caused confusion in the reports.
transactions
This was raised by POL's reporting systems.
2. I achaSé50L A user logged into a counter where there I Bug in the recovery process that could post transactions to the I No impact
wrong TP/ BP. This would result in 2 discrepancies in 2 separate
periods which would cancel out, so no long term impact on
Branches
Fix issued in June 2010, but in theory could impact counter that
migrated and hit this problem on the day of migration to Horizon
Online until September 2010 (when migration completed)
AC_153675905_1
45
POL00029012
POL00029012
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The stock unit was short by the amount of
the banking transaction in TP 12 BP 4, but
then had a matching gain in the following
period.
If the TP/BP is incorrect, but the stock unit
will roll into that period in the future, then
this problem will cause a discrepancy in
the current period, but it will be balanced
by an equal and opposite discrepancy in
the future, Advise the PM of this.
3. I acha633R
The Settle Gain/Loss Centrally products
have a minimum transaction value of £150
and should not be available if the
discrepancy to be cleared from Local
‘Suspense is less than £150. Horizon
Online does not appear to check the
minimum value when building the pick list
and so branches can choose settle
Centrally for any value.
This is an issue identified by Post Office rather than a branch in
that branches were being allowed to Settle Centrally small losses
(limit should be £150 or more). This was identified on 15" May
2010 and the fix rolled out to all branches by 5" July 2010
No impact
4. I AChambers253L
This was an issue picked up by the
reconciliation checks due to smart Post
not correctly checking that the pre-paid
amount for postage was less than the
actual amount and then attempting to
generate a postage label for a negative
amount
Unclear what branches would have done at the time, however the
impact is likely to have been very small (pence rather than
pounds). This was fixed in 2005.
Any impact would have been very minimal, pence
rather than pounds.
5. AllenD2519J
POLSAP report that a particular TC or TA
Back end problem, involving data sent to POL from a PO client.
No impact.
AC_153675905_1
46
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
is missing from the expected BLE file,
The TC / TA entry is actually present in the
BLE file, but it lacks the TC / TA
Reference value which allows POLSAP to
identify the item.
This is an issue caused by incorrect Reference Data with passing
data to POL's Back end systems (POL SAP) and may delay the
processing of a TA/ TC.
6. AllenD429U
One of the central systems failed during
the evening and when it restarted it picked
up the wrong time, which meant that when
it was trying to decide whether a Txn
happened before or after 7pm (the EOD
cut off) if got the answer wrong and this
meant that some data was associated with
the wrong day in back end systems.
Looks like a back end problem, with no impact on branches
This is a problem in the data Centre which delays passing data to
Post Office's back end systems. However, it has no impact on
Branch Accounts.
No impact.
7. ‘carde4027Q
This was a problem in incorrectly handling
a transaction which had been rejected by
the PIN PAD resulting in a spurious
reconciliation error on a report.
The issue first occurred in 2017 and another similar issue
occurred in 2013. It appears to be related to the old Hypercom PIN
Pads which were replaced after the second occurrence of this
issue so no further action was taken
As the transaction had been clearly declined, then there would be
no financial impact at the Branch
No impact.
8. carde5444K
The Postmaster received a Planned Order
as follows:
Based on the last declared Cash on Hand
figure of
£97,875.00 notes + £9,156.29 coins
on 13.07.2010 you will need to remit to the
Cash Centre
In a Shared SU, then it is possible to make multiple Cash
Declarations from different tills which are added together.
In this case a Cash Declaration had been made accidently for Till
16 (instead of Till 1) which resulted in the value of cash being
doubled.
‘Temporary impact due to human error.
AC_153675905_1
47
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
POL00029012
POL00029012
£85,000.00 in notes on your next
scheduled collection day.
The PM had declared cash for all his stock
units on that day, however his actual Cash
Declaration value was much lower than the
Planned Order cash declaration value. The
Planned Order asked him to remit too
much cash.
The PM normally used Till Id 1 for his
shared stock unit, but had accidentally
done a declaration with Till Id 16 during the
previous week. He was unaware that the
Till 16 declaration was being added to
every overnight cash declaration sent to
the cash centre.
This should have been spotted when the branch balanced.
However, before then the Cash Planning identified that the Branch
had too much cash and asked for some to be returned.
This is a case of branch user error and there was no actual impact
on the accounts, once the spurious Cash Declaration was
identified and removed.
AC_153675905_1
48