WITNO0780100
WITNO0780100
Witness Name: Richard William Roll
Statement No: WITNO078 0100
Exhibits: WITN0078_0101
Dated: 02 February 2023
POST OFFICE HORIZON ENQUIRY
FIRST WITNESS STATEMENT OF RICHARD WILLIAM ROLL
I, Richard William Roll, will say as follows,
1.
I joined the RAF in 1974 as an electronics apprentice and for 13 years I
worked on numerous aircraft navigation and weapons systems, some
analogue and some digital. I served in the UK, Europe and the Falkland
Islands. During this time I was sent on several computing and software
development courses, however the only qualification I had gained that was
recognised by civilian employers was a City and Guilds level three certificate
in electronics so I enrolled as a day-release student at Farnborough College
of Technology. Over the next four years I studied for a BTEC NC in
Engineering, followed by an HNC in Computer Studies. Following my
retirement from the RAF in 1989 I worked for about a year on factory
automation and robotics equipment, which included the programming of
Page 1 of 18
WITNO0780100
WITNO0780100
Programmable Logic Units (PLU’s) before leaving to take up a permanent role
as a programmer and developer (using the ‘C’ language) in 1990. A few
months later} IRRELEVANT
IRRELEVANT I
RELEVANT: For the next few years I worked part-time from home as a
programmer and business process analyst for various companies, becoming
proficient in various programming languages including Visual Basic, C/C++,
“IRRELEVANT
was able to take on more full time roles starting with hardware
installation and progressing to systems configuration and support; software
analysis, design, and coding; and project management. Immediately prior to
taking up the position at Fujitsu (technically, ICL — the company was
rebranded to Fujitsu in April 2002) I was working as a contractor at Eli Lilly
pharmaceuticals, managing and developing a global sales/CRM database
system.
2. I joined Fujitsu SSC (I can’t remember what SSC stood for) around March
2001 as a Product Specialist and held that position for three years, with no
promotion or change to my role, until I left in 2004 to attend university.
3. My role in Fujitsu SSC was to provide third line support on the Horizon
system. The system was hugely complex and included (amongst other
things) Windows NT systems and Unix servers; asymmetric digital subscriber
line (ADSL), microwave and satellite communications systems; and software
written in a variety of languages. Day-to-day tasks were many and varied —
for example, we may have been asked to examine database transactions in
Page 2 of 18
order to identify errors in the accounting system; examine computer
programmes to try to identify bugs; investigate communications failures in the
system; rectify failures in the overnight processing of the previous days
transactions; or liaise with engineers during the installation of new Post Office
counters. We also generated reports for senior management and the Post
Office, tested equipment, evaluated new hardware and software etc.
. When I joined Fujitsu, I received the same basic training on the Horizon
system as the SPM’s, although as I didn’t work in a Post Office I never
became truly familiar with the system operation from a SPM’s perspective. I
don’t recall there being any technical training as such — the system was so
complex and the role so demanding that you had to be an expert in one or
more fields when you started, you then picked it up as you went along. The
qualifications I had gained previously, my experience with different systems,
languages and in different roles, and my record of continuing professional
development — I was a member of the British Computer Society (MBCS), a
Certified Lotus/IBM Professional (CLP) and Microsoft Certified Systems
Professional (MCSP) — were considered valuable attributes for the
department.
. The department was headed by Mik Peach, with Steve Parker acting as his
deputy when required. We were all specialists in various areas — NT, UNIX,
accounting, programming — and work would either be picked up by whoever
was free or Mik would ask one of us to look at a problem. If one of us was
working on a problem outside our particular specialism and we became stuck,
then we would ask someone else more familiar with that aspect of the system
for assistance.
Page 3 of 18
WITNO0780100
WITNO0780100
WITNO0780100
WITNO0780100
6. The reporting structure within the department was very simple — we all
reported to the manager, Mik Peach.
7. Regarding the design and configuration of the system, my understanding is
that it was originally designed as a benefits payment system for the Benefits
Agency, but as costs spiralled they pulled out. Fujitsu had invested a huge
amount of time and money in developing the system and it would have been a
financial disaster if they couldn’t recoup some of their money; the Post Office
needed to update their counters; so the system was ‘sold’ to them. There
wasn’t enough time to redevelop the system properly, instead the Benefits
system was modified to try to meet the requirements of the Post Office. I
worked on one of the earliest iterations of the system; most of the counters
were connected by ADSL lines to the two main server farms in the north of
England, I don’t remember where exactly; some of the post offices had to use
other means of communication due to the limitations of the existing
communications infrastructure.
8. SPM’s would log in to the system at their post office (PO); there were several
different types of configuration, depending on the type of PO (mobile, single
counter, multi counter or main PO with a separate gateway computer), but
from the SPM’s perspective these would have all looked the same. General
security protocols were in place at the PO’s. Secure passwords were
required, which were not supposed to be shared but often were. There was a
secure link from the PO counter to Fujitsu’s servers — the counters would only
respond to requests originating from specific telephone numbers — and all
data was encrypted. Additionally, the network was completely (logically)
isolated form the internet, it had its own dedicated lines and for resilience
Page 4 of 18
these were duplicated, with one to the east of the country and one to the west,
I think (physically, the lines were probably shared with other BT customers).
However, due to geographical and financial constraints, these lines merged in
London and both were routed along one of the Tube lines; I believe there was
a minor train accident one day which unfortunately severed both cables and
left the north of the country isolated from the south.
9. We all had at least two PC’s at Bracknell; one ‘open’, which we used for
emails, researching the internet etc, and one ‘secure’ (completely isolated
from the ‘open’ system) for working on the Horizon system. Apart from
responding to requests for assistance from second line, for example, looking
into issues reported by SPM’s regarding accounting, product errors, hardware
failures etc, or queries from utility companies regarding payments made at
PO’s that hadn’t gone through, we also monitored the system and ran remote
programs we had developed which provided advanced warning of any
failures, for example with the overnight batch processing of network banking
transactions or benefits payments. This sometimes meant we had to connect
remotely to the SPM’s Horizon terminals, sometimes without their knowledge
or consent, to make changes to the counter configuration or the database
system.
10.1 think there were several ways to connect to the counters but it was a long
time ago and I can’t remember exact details. As I recall:
a. we could log into the Horizon servers using our own login
details and then use the RIPOSTE system to access the
Page 5 of 18
WITNO0780100
WITNO0780100
WITNO0780100
WITNO0780100
counters — any changes we made to the counter
database would then have our log in details attached;
b. we could log in through riposte another way, I can’t
remember the details, in which case it would be difficult to
see who had made changes;
c. we could go directly through the communications servers
to the PO gateway and then the counter — if the PM
wasn’t logged in then there would be no ID attached to
the database entries, which sometimes caused the batch
processing to fail overnight; if the PM was logged on then
any changes we made would have their ID attached - so
as far as the system (and any auditing) was concerned
the SPM would have been responsible for the
transactions.
11.Sometimes we were instructed not to let the SPM know we had altered his
system whilst he was logged in — to my recollection, sometimes POL
requested this, sometimes Fujitsu, and sometimes only our department knew
of it. It would have been possible for someone with the same access and
security privileges that we enjoyed in SSC, to log in to a PO counter and
transfer money to an external bank account without the SPM being aware of
this. I do not know if anyone else in Fujitsu had such privileges.
12.1 worked on a very early version of the Horizon system, there was talk of
moving to a web-based version but I don’t remember much about this. I
believe that as time went on, the auditing process improved and personnel
Page 6 of 18
WITNO0780100
WITNO0780100
could no longer access the counters anonymously, as and when they wanted
to.
13.1 have been asked by the Inquiry to describe how the Horizon system was
used by Sub Postmasters, but I cannot remember any details regarding this,
other than that they logged on to the system and keyed all of their
transactions through it.
14.1 have been asked by the Inquiry to describe how the Horizon system was
connected from Fujitsu to Sub Postmasters, I remember it used various
communication channels such as ADSL and satellite but I can’t remember any
other details.
15. The Inquiry has asked about security protocols regarding access to SPM’s
systems. I don’t remember any security protocols; we sometimes connected
to PO counters without the postmaster being aware that we were ‘looking over
their shoulder’. In the early days I frequently logged on to counters to see
what was happening; there was no record of my doing so but I think this
changed after I had left.
16. The Inquiry has asked about any risks that arose because of any access to
individual systems. There are always risks when logging into a live system,
there was a mistake made once that had a huge impact on the SPM’s
accounts/database but I don’t remember any specific details.
17.In my opinion the coding and development of the system did not meet my
expectations of quality for a major software project; I considered it to be a very
poor system that should never have been deployed but I cannot be more
specific than this.
Page 7 of 18
WITNO0780100
WITNO0780100
18. The Inquiry has asked if I can describe the structure of the communication
between Fujitsu and the SPM’s but I cannot remember any details.
19.The Inquiry has asked about my understanding of the levels of support offered
to SPM’s but I can’t remember.
20. The Inquiry has asked about how any service level agreements affected my
role; there was pressure to fix problems or provide an answer before SLA time
ran out, on at least one occasion I was emailed by my manager asking when I
could provide an answer/solution because we were nearly out of SLA time,
but that is all I remember.
21.The Inquiry has asked how issues made their way to me, how I received the
information, who gave me the information and who had dealt with the query
before it came to me but I don’t remember any details.
22.The Inquiry has asked me to describe the types of defects that I dealt with but
this is a hugely complex question and I don’t remember any more details than
I have previously described. All I can say is that if anything at all went wrong
with any of the hardware or software, anywhere in the system, SSC might be
asked to look into it. Whatever the problem was, if no-one else could fix it we
would be called in.
23.The Inquiry has asked me to describe the steps I would take to investigate an
issue but I can’t remember any specific details. There were so many types of
issue; sometimes I spoke to the SPM, sometimes I connected to their Horizon
Terminal, sometimes I had to get the equipment recovered back to Bracknell
so I could strip it down and work on it in-house, sometimes I spoke to BT/utility
companies regarding the post office, sometimes I spoke to third parties
Page 8 of 18
WITNO0780100
WITNO0780100
regarding BACS payments. At times I got called out over night to fix problems
with the UNIX servers, or worked with engineers to set up new post offices, or
with the testing or development teams to investigate potential programming
bugs.
24.The Inquiry has asked about contact with SPM’s; I spoke to SPM’s every
week but I can’t remember any of the details.
25.The Inquiry has asked about the information used to review an issue. That
depended upon the issue — with a counter, or a server, or with the
communications systems. There was a vast array of data available but I can’t
remember anything more specific than this.
26.The Inquiry has asked how long it would take to find a solution, but that all
depended on how difficult the problem was. It could take a few minutes, or it
could be that we identified a problem and passed it on to another department
for a fix, in which case it could be months before it was resolved.
27.The Inquiry has asked if there were any deadlines for finding a solution, but
that depended upon the problem. There were SLA’s for some parts of the
system but not others, and there was pressure to meet these deadlines, but I
don’t remember any details.
28.The Inquiry has asked how often I successfully resolved an issue; I should say
almost always but I can’t remember any details. I’m sure that on occasion I
wasn’t able to come up with a solution.
29. The Inquiry has asked how long it would take to roll out a solution, again that
depends on the problem. Sometimes an error could be corrected in minutes,
Page 9 of 18
WITNO0780100
WITNO0780100
other times it could be months before the software could be developed and
rolled out, but I can’t be more specific because I don’t remember.
30.The Inquiry has asked how wide the solution roll out would be; that depended
upon the problem, it could have been one transaction on one counter, or a
database update to 40,000 counters; or it could have been the servers at the
server farms and not on the counters at all.
31.The Inquiry has asked whether there were any difficulties in rolling out a
solution; I’m sure at times there were but I can’t remember any details other
that that on one occasion a software patch was rolled out to fix a bug, but the
developers were writing new code and when this was rolled out it undid the
fix, so we suddenly started seeing a problem that we thought had been
resolved months before; and another time someone released the wrong
security key and 25% of the estate went off-line before we could stop it.
32.The Inquiry has asked me to describe the circumstances in which I had
contact with SPM’s and any difficulties that arose but I can’t remember any
details, other than that due to cultural differences some SPM’s refused to
speak to the female members of the team.
33.The Inquiry has asked me what records were kept of the issues that arose
and how that information was used but I can’t remember any details.
34.The Inquiry has asked me what happened if I was unable to resolve a defect.
That depended on the type of defect and I can’t remember any details, other
than that if it was an accounting problem then if we ran out of SLA time I think
we reported that we couldn’t find the source of the problem. All hardware
Page 10 of 18
issues were, to the best of my knowledge, resolved speedily; and I think all
issues with overnight batch processing were resolved within the SLA periods.
35.The Inquiry has asked if the situation was different if I needed more time to
investigate the issue but I can’t remember what the process was. If we were
stuck we sometimes got more members of the team involved to try to resolve
the issue asap, before we ran out of SLA time.
36.When we had finished with a problem, either by providing a solution or by
being unable to find or duplicate the reported problem, then I think we
informed our manager (Mik Peach) but I can’t really remember any more than
this.
37.The Inquiry has asked what the SPMs were told but I do not know what
information was communicated to SPM’s.
38.The Inquiry has asked what involvement the management of my department
had in dealing with issues that could not be resolved but I don’t know, I was
never involved in this.
39.The Inquiry has asked if any records were kept of the issues that could not be
resolved but I can’t remember.
40. The Inquiry has asked if I can describe any issues that arose as the Horizon
IT system was developed to handle more products but I can’t remember.
41.The Inquiry has asked me to describe how these issues were handled and
whether, in my opinion, this was sufficient but as I can’t remember any issues
then I can’t answer this question.
Page 11 of 18
WITNO0780100
WITNO0780100
WITNO0780100
WITNO0780100
42.The Inquiry has asked if I can describe any issues with the hardware provided
to SPMs to operate the Horizon IT system and how were any such issues
dealt with. I only remember one issue with hardware, where an SPM seemed
to be switching her laptop off before 6.00pm. Because the power was
switched off the Laptop could not generate the end of day financial markers,
consequently the accounts were not processed correctly which resulted in
transaction data not being sent to the banks and utility companies so Fujitsu
missed SLA deadlines. I was asked to investigate; the SPM (a very
experienced lady) insisted she was not turning the machine off but the log files
on the counter showed that the laptop was being powered down. I arranged
for the laptop to be swapped out and returned to Bracknell for testing and
found that when the screensaver button was pressed the power to the
machine was switched off. When I disassembled the machine I discovered
the fault — during the build the wires had been cross connected.
43.1 brought this to my manager's attention as I felt it should be investigated
further; a few days later he called me over and informed me that the manager
of the section that assembled the Laptops knew about the issue already as
one of his engineers had told him that he had inadvertently mis-wired several
laptops that had been sent out to SPM’s. I was told that no further action was
to be taken and I was instructed to record the fault as no fault found or
something similar — the incident was hushed up without senior management
or the Post Office being made aware of it. The faulty laptops remained in
general circulation, but as none of the other SPIM’s used the screensaver
button regularly it did not cause a problem.
Page 12 of 18
WITNO0780100
WITNO0780100
44. The Inquiry has asked if I consider that the issue described could have been
avoided and how. I believe that there were quality control issues within Fujitsu
— if the Laptops had been properly tested before being dispatched then this
problem would have been avoided. Although the employee concerned alerted
his manager to the problem, his manager had no idea what the consequences
could or would be so decided to ignore the problem. This particular issue was
hushed up by my manager because the other manager was a friend of his.
45. However, this raises questions regarding communication, honesty and
transparency within and between departments within Fujitsu. For example,
mistakes were made when releasing updates to the software and it is feasible
that a programming error could have been rolled out to the estate and a fix
rolled out a few days later, without anyone in the wider organisation being
informed. In this scenario, if an SPM had problems with their accounts then
by the time SSC were asked to investigate the fault would have already been
rectified so we would not have been able to duplicate the error. It would have
seemed that the only logical explanation was that the SPM was to blame, with
potentially catastrophic consequences for that individual.
46.The Inquiry has asked if any improvements were made in relation to the
handling of Horizon IT issues; I am sure they were but I can’t remember any
details.
47.The Inquiry has asked if during my time at Fujitsu, I was concerned about
management practices in relation to unresolved defects; to the best of my
knowledge I was unaware of any management practices outside of my
department other than the examples I have previously described. There were
Page 13 of 18
WITNO0780100
WITNO0780100
rumours of historical instances of intimidation and bullying in some
departments but at the time they seemed utterly fantastical and unbelievable,
however I have since heard other stories which corroborate the stories I was
told by various members of the team.
48.The Inquiry has asked if I can describe any consequences for Fujitsu if
defects were not resolved. I believe that there were SLA penalties which
could have potentially cost Fujitsu millions of pounds but I can’t remember any
details.
49. The Inquiry has asked if I was aware of any other wider problems at Fujitsu; I
believe that Fujitsu was in a very poor position financially and that the Horizon
project was all that was keeping the company afloat at that time (during my
time at Fujitsu, there was a pay freeze and numerous cut
backs/redundancies) consequently, the system had to be portrayed as
working almost perfectly. Incidentally, SSC were awarded the President's
Award in 2003 for our contribution to the company.
50. The Inquiry has asked if I was aware of meetings taking place between Post
Office Limited and Fujitsu during my time there; I was not. Occasionally
visitors would be shown around the SSC but I don’t recall being told who the
visitors were.
51.The Inquiry has asked if I was aware of meetings taking place between the
government and Fujitsu; I was not.
52.The Inquiry has asked if I was aware of meetings taking place between any
Members of Parliament and Fujitsu; I was not.
Page 14 of 18
WITNO0780100
WITNO0780100
53.The Inquiry has asked me to describe the circumstances in which I became
concerned about the issues with the Horizon IT systems but I can’t remember.
I think I heard something on TV or Radio or read something in a magazine,
maybe around 2012, so I contacted someone who had been involved and
said I had worked on the system and I was willing to talk about my time there.
That is all I remember.
54. The Inquiry has asked if I am in possession of any correspondence or other
documentation from my time working on the project; I am not.
55. The Inquiry has asked if I there any other information I am able to provide
which is relevant to the Inquiry. I have used PowerPoint to create a basic
Schematic Diagram showing how the Horizon System worked, and how SSC
could access any part of the system (WITN00780101).
Statement of Truth
I believe the content of this statement to be true.
02 February 2023
Page 15 of 18
WITNO0780100
WITNO0780100
Page 16 of 18
WITNO0780100
WITNO0780100
IRRELEVANT
Page 17 of 18
Index to First Witness Statement of Richard William Roll
No.
URN
Document Description
Control
Number
WITNOO78 0101
Basic Schematic
Diagram showing how
the Horizon System
worked, and how SSC
could access any part of
the system
WITNOO78 0101
Page 18 of 18
WITNO0780100
WITNO0780100