1
2.
WOMBLE
BOND
WBONO000193
WBON0000193
DICKINSON
RESPONSE TO RICHARD ROLL (TO BE TURNED INTO A STATEMENT BY STEVE PARKER)
References to paragraph numbers are to paragraphs of Mr Roll’ statement.
I worked with Mr Roll while he was employed by Fujitsu. Although I did not have the formal title, I
acted as team leader.
I found Mr Roll to be a conscientious worker and provided him with a reference for a position that
he applied for after leaving Fujitsu. As described below, he was not what I would describe as a
technical person and he was mainly tasked with undertaking standard work arounds [please
provide some examples]
As Mr Roll was employed between 2001 and 2004 his references to Horizon are to the original
Horizon system (sometimes referred to as Legacy Horizon) as opposed to Horizon Online/HNG-
X. When deal with matters in this statement I am describing the position between 2001 and 2004
unless otherwise stated
[have reviewed paragraphs [xx] of the witness statement provided by my colleague Torstein
Godeseth which describe the core audit process in Legacy Horizon and I confirm that they are
accurate to the best of my knowledge and belief] [TBC]
Structure of Fujitsu's support teams for the Post Office Account
6.
6.4
62
63
64
There were four lines of support for Horizon while Mr Roll was employed by Fujitsulis this still
the position?):~
[FJ - please can we flesh out the descriptions of the four levels of support? How many
people are in each group? Do they follows stated processes to ensure quality and
consistency of delivery? Is the setup industry standard?]
1* line: the Horizon Service Desk, a helpdesk that branches may contact with issues relating to
Horizon. The helpdesk deals with straightforward issues such as password resets and refers
other issues to the 2" line support team;
2" line: search knowledge articles and apply simple, well-defined work-arounds (often on the
phone) such as guiding branch staff to return to the Horizon home screen if they get stuck in part
of the system [someone mentioned monitoring systems on our call - can you expand on
that please?);
3° line: write the knowledge articles used by the 2" line support team, carry out fixes in relation
to configuration type issues (as opposed to software fixes, an example being where the system
crashes while [when?] and the branch required assistance to unlock it) and examine some code
[for what purpose?I; and
4" line: system development and software fixes [can we describe the qualifications required
to be a member of 4" line and their calibre?].
The 3* line support team is only ever engaged when there has been a problem. [Insert section
re metrics around various I 8;
and less to 3rd line. Metric
‘A mature system now, but still worth saying that you have to bear this thing in mind.
Need figures from Sandy Borthwick (HSD days). PN/DI to chase up.)
Mr Roll’s role
Mr Roll was a member of the Software Support Centre (SSC) which primarily provides the 3° line
‘support (there can be some overlap between the four lines of support and SSC does provide
some 2° line support)
AC_182319934_1 1
‘Commented [SH]: In order to justly the conclusion that
Roll was not technical (which is important) we need to be
specific about what he did, where this fitted into the overall
team and why others were far more engaged in technical work
than him. Are we trying to make any further point here i.e. that
‘a8 a result of his role he would not have understood aspects
‘on which he is giving evidence?
‘Commented [SH2]: Worth making the point somewhere
that there isa degree of overlap between the various lines of
support ie. not clear cut distinctions
WBD_000063.000001
As with any mix of people, there are various levels of talent within SSC. Mr Roll was primary
utilised in Operational Business Change (OBC), which involved supporting the engineers who
were opening and closing branches and also increasing and decreasing the number of counters
in branches and replacing failed counter hardware. As part of that role he may have spoken to
Postmasters from time to time. [Can we describe the sorts of things he would not have
routinely seen as part of his role? Is there a possibility that his witness statement covers
things which he would not have had first-hand knowledge of? If so, please provide
examples.]
Mr Roll was not involved in the provision of 4" line support. Some members of the 3" line
support group (but not Mr Roll) identify the need for software fixes and pass this on to the 4" line
team for a code fix to be written. Such a code fix would not be written by anyone in the 3” line
team.
Branch discrepancies
"1
15.4
Mr Roll states that "software issues" which "could, and did, cause financial discrepancies at
branch level, including "shortfalls" being incorrectly shown on the Horizon system” were
"routinely" encountered (paragraph 10). Its true to say that as with any system, software issues
in Horizon may cause the illusion of a discrepancy in a branch's accounts from time to time.
However, it does not happen *routinely’. (can we give an anecdotal estimate of how regularly
such issues arise?I Further, in the vast majority of cases such an occurrence would cause a
receipts and payments mismatch that would be flagged [by who?] and investigated [by who?]
[Wy most bugs? What type of bug would net cause a receipts and payments mismatch?
How would they be identified?)
[Explain KELs and Peaks/how errors are identified and spotted? Note GJ's comment that it
ifficult" to check how many issues caused discrepancies in Old Horizon, but FJ
is tracking things better in new Horizon" (e.g. the three known bugs)]
Post Office operates its own 1* line helpline for branches called the National Business Support
Centre (NBSC). Ifa branch requires assistance to attempt to determine the cause of a
discrepancy they may contact NBSC. However, Horizon records the data that is entered into it
by branch staff and that may not reflect what actually happened in the branch. For example, a
user may enter a deposit of £100 into a customer's bank account on Horizon but rather than
taking £100 from the customer, they may make a mistake and give the customer £100 as if it had
been a withdrawal. Therefore, NBSC could not always help branches to identify the cause of a
discrepancy and they would sometimes ask SSC to assist. One of the reasons for this is that
‘SSC has access to transaction data whereas the NBSC does not. Mr Roll’ statement that "/iff an
error was referred to us then it was extremely unlikely to be due to a mistake made by a
postmaster” is simply not correct. [Can we say roughly what proportion of issues that made
their way through to SCC between 2001 and 2004 would have been due to software issues
and how many were presumed to be human error because no other cause could be
found?]
Mr Roll suggests that he would investigate financial discrepancies that had arisen in branches by
"work{ing] sequentially through all transactions over the relevant period, and also work{ing]
through thousands of lines of computer coding’ (paragraph 7). To the best of my recollection Mr
Roll would not have worked through thousands of lines of computer coding to investigate a
discrepancy in a branch. While SSC has access to Visual Basic (VB) code [what is this code
and what other types of code exist that SSC did not have access to7], it was rarely used
[what purpose was it used for? Can we explain why RR would not have needed to look at
it?]. Mr Roll would follow work around processes designed by other people and was never a
detail person[ which would not require him to review VB code - is this correct?].
Mr Roll states that "/sJoftware programs were written by us to strip-out irrelevant data, to enable
us to more easily locate the error.” I am aware of two programmes that were written while Mr Roll
was employed by Fujitsu:-
the Smiley program written by my colleague John Simpkins which [describe what it did]; and
AC _152310534_1 2
WBONO000193
WBON0000193
WBD_000063.000002
15.2
the [insert name] programme written by my colleague Richard Coleman to extract messages
from the correspondence server to local text files for examination.
[What errors is Mr Roll talking about in para. 7 of his statement and how did these
programmes help to spot them?]
I do not accept that SSC "regularly" identified issues with the computer coding. As with any
application, there were some coding errors in Horizon. A lot of the stuff we saw was [misuse of
the system, actions to ensure that dataset system was using was complete][please expand on
this]
Mr Roll states that if SSC was "unable to find the cause of the discrepancy then this was reported
up the chain and it was assumed that the postmaster was to blame" (paragraph 10). That is not
my experience: SSC feeds factual data back to Post Office and might say something along the
lines of "all indications are that the branch has made a mistake" if there is no other explanation,
but SSC does not attribute "blame".
‘SSC between 2001 - 2004
19.
20,
a
22,
23,
I agree with Mr Roll's recollection that there were around 30 individuals working on the 6" floor in
Bracknell at any one time during this period. [How many were part of SSC? How is the 30
split?] However I strongly disagree that much of the work being carried out could be described
"as "fire fighting" coding problems in the Horizon system.” There would be times when SSC
would be firefighting when, for example, a data centre went down. However, as described above,
‘SSC would primarily provide 3” line support including [writing the knowledge articles used by the
2° line support team, carrying out fixes in relation to configuration type issues (as opposed to
software fixes, an example being where the system crashes while [when?] and the branch
required assistance to unlock it) and examining some code.
[Can we give some anecdotal evidence as to the proportion of software issues that could
create the illusion of a branch discrepancy?]
There were Service Level Agreements for issues such as stuck transactions (Fujitsu had 10 days
to retrieve transactions that had got stuck in a counter). I do not understand what Mr Rolls,
means when he says that "any discrepancy in the post office accounts had to be resolved
speedily’ (paragraph 12). [Is this correct or do you know what he means?]
While it is correct that there was a limited number of opportunities to release software updates
and it could be six weeks before a fix could be released, but in the vast majority of cases it did
not take six weeks to do this. I do not understand what Mr Rolls means when he suggests that a
bug could reappear several weeks after a coding fix had been released due to software issues
(paragraph 14), although I am aware of cases where a fix regressed [how many times has this
happened?].
[What proportion of issues are: (1) software; (2) hardware?]
Remote access
24
25,
26,
Mr Roll claims that "/dJuring the course of resolving the software issues, we would frequently
access a Post Office counter IT system remotely" (paragraph 15). By remotely I assume that Mr
Roll means accessing Post Office counter IT systems while not in a branch
The first point to note here is that a level of such access was required in Legacy Horizon because
itwas a distributed system (i.e. [please explain what this means]). This means that such
access was required in order for Fujitsu to support users.
The example given by Mr Roll ("when a binary bit would “fip’, thus a "1" became a "0™) relates to
a flag which can become locked in the wrong binary setting (1, 0), preventing updates to stock
units within a branch. This does not involve accessing or editing transaction data in any way or
re-creating databases. The tool used to correct this issue is described in [exhibit
AC _152310534_1 3
WBONO000193
WBON0000193
WBD_000063.000003
27.
27
272
273
28,
29,
29.1
29.2
30.
31
32.
32.4
32.2
DEV/APP/LLD/0202]. [I understand from Post Office's solicitors that this document has been
provided to the Claimants as part of these proceedings.]
Where Mr Roll refers to re-creating the database I understand him to be referring to the process
that Fujitsu would follow if data became corrupted [how often would this happen?]. As
explained by Mr Godseth in paragraph 35 of his statement, in Legacy Horizon:-
all counter data was held in a bespoke message store’ and the data was replicated within each
branch to aill counter positions and from each branch to the data centres where it was held in the
correspondence server message stores;
any data inserted into the message store at the data centre (for example reference data or
authorisations for banking transactions) would be replicated back to the branch counters; and
selected data was then extracted from the correspondence servers to update Post Office's back
end systems.
If one of the sets of data became corrupted [how would FJ know it had been corrupted?] on a
counter SSC would delete the entire set of data on that counter and replace it with a copy of the
data from one of the other sources that had not been corrupted. While this process involves
deleting and replacing a set of data, no new data is produced; all that happens is that the correct
data is used to replace the data that has become corrupted. [It would have been necessary for
SSC to inform a branch before carrying out this task.] [why? would it have caused a problem if
the branch tried to use Horizon while this was happening?]
Mr Roll also claims that:~
"some errors were corrected remotely without the sub-postmaster being aware" (paragraph 16);
and
there were some errors where it was necessary to "download and correct the data and prepare it
for uploading back on to the post office computer, then call the postmaster to inform him that
there was problem and that we needed two or three minutes to correct it" (paragraph 17)
Itis not clear what errors Mr Roll is referring to or how he says they were corrected. As
explained by Mr Godseth in paragraph 36 of his statement, users with sufficient access
permissions could inject additional messages (i.e. data) at the correspondence server and the
fact that such data had been injected in this way would be clearly identifiable. [Note - GJ now
says that SSC could inject data at the counter. Need to bottom this out ASAP and explain
whether such injections were clearly identifiable] [Need to say whether or not these things
could be done without informing the branch]
It may be that Mr Roll is referring to issues relating to the end of day concept in Legacy Horizon,
Essentially there was a cut-off point for transactions every day [what time?] and each counter
had to write an end of day message to the branch's master counter to enable the master counter
to harvest messages (including details of transactions) to Post Office's back end systems.
Occasionally a counter in a branch would fail to write an end of day message and there was a
process for correcting it. The issue would be reported to SSC by way of a Peak (either as a
result of a call to HSD or sometimes FJ could spot issues via system events). In lay terms, SSC
would force the counter to generate a report based on the data already in the counter; this would
not alter the transaction data. A message injected in this way would go into the audit trail
{in summary:-
Fujitsu could not change transaction data; and
Fujitsu could inject transaction data and such injections were clearly identifiable.] [Why would FJ
want or need to inject transaction data?}
* A message means data and transaction data is a subset of the data in the message store.
AC _152310534_1 4
WBONO000193
WBON0000193
WBD_000063.000004
WBONO000193
WBON0000193
33. I:do not understand what Mr Roll means when he says that SSC had the abilty to “transfer
‘money remotely” (paragraph 18). Nor do I understand how a third party could have accessed the ‘Commented [SH3]: Need a more robust rejection here.
system and I note that Mr Roll has not attempted to explain this (paragraph 18). It was not (and The position, as I understand i is that Fd could not do
is not) possible for Fujitsu to do anything with money in a branch. [As noted above, the only thing Fede eel ate el ride mea bend
i , transfer it into or out of a branch and this needs to be
they Fujitsu could do in theory was inject transaction data and it would only do so [insert fedaioels pe erty Gog ney coll err siacty Waa ee
explanation]. [Could FJ in theory manipulate a branch's transaction data in a way which transaction data (not amend existing transaction data), that
was detrimental to a particular SPMR and undetectable is simply wrong.] there was no point in doing s0 (ie. it was something they
‘would only very rarely want or need to do) and that it was
‘anyway a pointless exercise since t would immediately be
picked up by Horizons checks. The key poin, which think
does nol yel come across inthis drafts tha the suggestion
that FJ was able to manipulate a branch's transaction data in
‘a way which was detrimental toa particular SPMR and
Undetectable is simply wrong.
AC 152310594. 5
WBD_000063.000005