WITNO09570100
WITNO9570100
Witness Name: Rupert Lloyd Thomas
Statement No.: WITN09570100
Dated: AL May 2018
POST OFFICE HORIZON IT INQUIRY
FIRST WITNESS STATEMENT OF RUPERT LLOYD THOMAS
I, RUPERT LLOYD THOMAS, will say as follows...
INTRODUCTION
1. I am a former employee of Post Office Counters Limited ("POCL") and held
the position of Information System Specialist.
2. This witness statement is made to assist the Post Office Horizon IT Inquiry.
BACKGROUNDIINITIAL CRITICISM OF ESCHER'S RIPOSTE SYSTEM
3. I worked at POCL during and after the Horizon procurement period and left on
1 August 2001.
4. The Horizon project was at odds with POCL policy and culture. Outsourcing a
mission-critical system to a third party had never been tried before.
5. At the time, POCL staff were being urged under Total Quality Management to
look outside of the business for best practice, to benchmark, to seek excellent
exemplars, align with other organisations to benefit from their experience, and to
Page 1 of 11
WITNO9570100
WITNO9570100
use industry standards. The aim was to enter POCL for a Baldridge
Quality Award.
. The question was whether POCL should rely on software that had already
been in use by other retailers or banks, with a tried and tested track record,
or bespoke-software created by the Horizon project.
. The POOL Information Systems Strategy Unit ("ISSU") staff believed it was
possible to use an existing Retail EPOS system as the basis of the new Counter
System, using proven code and benefitting from problems solved elsewhere.
Why take on the burden of responsibility of a bespoke development when the
Post Office, and the public sector in general, had a dreadful track record in the
field?
. There were two opposing groups within POCL. The more traditional group
found it hard to accept that counter transactions were similar to those in retail
and banking and saw little benefit in looking outside. They could not see that
selling a book of stamps was no different from selling a can of beans. They
were firmly in the bespoke camp.
. The main advocates for the use of an existing Retail EPOS system were the
staff in the ISSU at headquarters who consisted of Basil Shall, Wendy
Powney, Charles Hooper, and me. Charles Hooper had been a member of
the procurement assessment team and had seen that other suppliers were
tendering existing EPOS systems at the core of their solutions.
Page 2 of 11
WITNO09570100
WITNO9570100
10. There was deep unease in ISSU when the procurement selected the ICL
consortium which contained an immature and unproven bespoke system —
the Riposte System from Escher. Escher had no experience in large systems,
let alone a project on the scale of Horizon. I visited Escher in Dublin, a small
operation with a tiny market share. Relying on Escher was akin to gambling.
11. The ISSU knew that the ICL Retail Systems (a different division of ICL
from ICL Pathway) had acquired Post Software International ("PSI") and its
GlobalSTORE product on 10th June 1996, before the Horizon
procurement. ICL did this to strengthen their retail offer.
12. There was every opportunity to ‘ride both horses’, but POCL left
itself without a Plan B as ICL Pathway suffered from lengthy delays.
13.A derivative Of the ICL GlobalSTORE EPOS system is still for sale
from Fujitsu today in 2023.
14. GlobalSTORE conformed to many retail technology standards which would
have aligned POCL with other retailers and provided access to new retail
developments for example, payment cards, special offers etc. Using a known
EPOS retail system would provide access to regular software updates and
sharing costs with other users. PSI GlobalSTORE conformed to the Association
for Retail Technology Standards ("ARTS") data model. The ARTS organisation
in the USA is still active today in 2023. This means decades of lost opportunity.
15. Adopting standards would help POCL control the supplier, and this
was well known in the 1990s.
Page 3 of 11
WITNO09570100
WITNO9570100
16. The debate known as "make versus buy" went on from March 1996 to at least
August 1997 but was effectively over by 11 September 1996 when Paul Rich
(a marketing man with no IT experience) emailed Charles Hooper et al to reject
GlobalSTORE approach. He wrote "...to introduce a PSI solution at this stage
would be dangerously risky to the timetable. Andrew (Stott) and others
estimate a delay to release 1 of up to a year is likely." "...so, my decision is to
stick where we are."
17. Although Paul Rich had already ruled out using GlobalSTORE, he visited PSI in
California for a presentation of their GlobalSTORE product on 7 October 1996.
He was accompanied by Bob Peaple (IT Director with no IT experience), Byron
Roberts (Operations, no IT experience) and Wendy Powney (IT but a junior
member).
18. The ICL "makers" continued with their bespoke solution which aligned with
nothing, conformed to no standards, was delayed until 1999, and led to disaster.
19. I wrote a timeline of events, dated 22 May 1998, (WITN09570101), as I felt
the senior managers at POOL were making unwise technical decisions, which
they were not qualified to make, and were no longer listening.
20. There was enough doubt about the ICL Riposte approach to warrant
spending money to send a team of POCL managers to California to see
an alternative existing retail EPOS system.
Page 4 of 11
WITNO9570100
WITNO9570100
POTENTIAL REASONS FOR INTERMITTENT HORIZON FAILURES - SOFTWARE
21.
According to,a Computer Weekly inside source, the coding had "no design
documents, no test documents, no peer reviews, no code reviews, no coding
standards." The Horizon software was what is known in computer parlance as
a kludge. Sloppy, non-conforming software with bugs were identified as the
root cause of the errors, but we should ask why were there problems in some
places and not others? A contrarian would say Horizon worked in most places
and the total losses were not sufficient to alarm the accountants.
POTENTIAL REASONS FOR INTERMITTENT HORIZON FAILURES - HARDWARE
22.
23.
24.
25.
There were reported lockups and freezes and Microsoft Windows
NT blue screens. I think these issues were disastrous.
Microsoft Windows NT was introduced in 1993 and Microsoft in the 1990s
were new to the retail EPOS market. By 1996 Horizon was betting the
farm on Microsoft Windows NT and this would be on the risk register if it
exists? Was Microsoft Windows NT ready for retail?
Other questions to consider are: Was Microsoft Windows NT being updated with
fixes? Was the hardware being damaged in the rugged retail environment due to
crashes and inelegant shutdowns? Was the hardware subject to hard reboots?
I had a Microsoft Windows NT office system at the time and was familiar with the
blue screen of death ("BSoDs") — a stop error. Accbrding to Wikipedia, BSoDs
Page 5 of 11
WITNO09570100
WITNO9570100
can be caused by poorly written device drivers, malfunctioning hardware such
as faulty memory, power supply issues, overheating of components, or
hardware running beyond its specification limits. A BSoD will produce a stop
error code which points to a particular problem. Were these being recorded and
analysed in Horizon?
26. It was unreliable hardware and operating system.
POTENTIAL REASONS FOR INTERMITTENT HORIZON FAILURES — NETWORK
27.In 1996, the largest retail network in the UK was the LINK network that facilitated
ATM sharing. Getting LINK to work reliably had taken many years and
compared to this, the RIPOSTE messaging capability was new and immature.
Managing financial transactions over fragile and unreliable networks took
experience and great effort.
28. It was being rolled out in Eire but was not fully proven at the large-scale in
POCL. If the RIPOSTE software could not manage, in a robust fashion,
network dropouts and brownouts, it was likely that Post Office transactions
would get "lost in the network" or, worse still, would get overwritten.
Stories from Subpostmasters ("SPMs") tell of counter totals changing without
human intervention. This has been seen as the Fujitsu staff making changes
remotely, but why did the errors occur in the first place? That may be because
of badly implemented software by Fujitsu but may also be because of faulty,
immature, and unproven communications software written by Escher.
Page 6 of 11
POTENTIAL REASONS FOR INTERMITTENT HORIZON FAILURES - POWER
29.In the mid-1990s, the previous counter system at the Branch offices, known
as ECCO, was suffering from chronic failures, and plagued by data loss.
POOL by that stage already had a lamentable record in losing data. This
was attributed to what became known as a "sector slip".
30.A sector slip was writing the transaction data from the EPOS terminal to the
wrong part of the floppy disc (the hardware being an inherently cheap solution)
thereby ruining the session for the clerk at the counter. They had to punch in all
the transactions from that session again, from paper records. Offices had
cupboards full of ruined floppy discs. There were terrible rows involving myself
trying to get something done about it. The attitude was we were getting the
new Horizon system so why waste money on the old one?
31.A senior technical manager at IT Paul Santilli said: "We may never find
the cause." I said the cause would not be found unless we looked for it.
32. There were critical lessons to be learned with an existing operational
system, which could be. applied to Horizon. There was no culture of
learning from mistakes.
33. Following a meeting in La Jolla, California, between Bob Peaple and Byron
Roberts, in October 1996, it was eventually decided to appoint Jack Kirk from IT
to look into the matter. Kirk concluded that error trapping in the software
attempted to write the transaction three times, then if it failed, put the transaction
in the wrong place on the disc in error. £500,000 was spent to uncover this.
Page 7 of 11
WITNO09570100
WITNO9570100
34. The ECCO system, managed by the POOL Operations team, had no known
protection against electrical outages, power spikes, brownouts or drops. Desk
research at the time showed that other retailers had experienced power
problems and done something about it. POOL did not know whether it had a
problem or not. It is not likely that POOL was immune to power issues and the
use of battery-backup for computers was at the time over fifty years old.
35. Byron Roberts, a senior POCL operations manager, refused to investigate
the power issue. Some unauthorised very small-scale investigations (one
Uninterruptible Power Supply machine), in collaboration with American
Power Conversion, a supplier, proved inconclusive.
36. The remedy was known about at the time, but nothing was done. The
lessons were not applied to the Horizon project.
37.No Uninterruptible Power Supply (UPS) was fitted in the offices which would
have prevented damage to the EPOS terminals. Guarding against crashes
was crucial to the smooth operation of the system. Every effort should have
been made to prevent them. Prevention is better than cure. The crashes
should not have happened.
38.From the deposition of Pam Stubbs, we know that she endured 36 power
cuts in one day. You don't need any technical knowledge whatsoever to
conclude this might be the root cause of her difficulties.
39. The equipment at Louise Dar's office may very well have been damaged
before she took over. Why did it take two hours to boot?
Page 8 of 11
WITNO09570100
WITNO9570100
WITNO09570100
WITNO9570100
40.Pam Stubbs says when they closed 3,000 offices the equipment was
repossessed for use elsewhere. SPMs were receiving used equipment with
an unknown track record, where they could not possibly know what had
happened to it previously. Was proper testing conducted on the used
equipment prior to delivery? Were POCL passing on problems?
41.Was any technical advice issued to SPMs about avoiding connecting electrical
devices to the EPOS circuit e.g. don't plug in a kettle? Many stores would be in
a hostile electrical environment, chest freezers cutting in and out etc.
42. There was inadequate power protection.
POTENTIAL REASONS FOR INTERMITTENT HORIZON FAILURES -
CULTURAL ISSUES
43. The local managers who were responsible for the Sub Offices would in
many cases have had very little knowledge of computer systems and would
have struggled to make judgements about any glitches that occurred in
Horizon. For many this would have been beyond them. We know they ‘went
along’ with punitive measures against SPMs.
44.To say that a SPM should determine whether Horizon was operating correctly is
not feasible, particularly if the reporting suite did not support this? The onus was
on POCL to supply a system that was fit for puppose. Could the SPM print out the
familiar cash account in Horizon? A document that offices had relied on since the
Page 9 of 11
WITNO09570100
WITNO9570100
year dot. Was the introduction of the new computer system accompanied
by unfamiliar procedural changes?
CONCLUSION
45. The subsequent position of POCI.. was that Horizon worked satisfactorily in
most locations, and that errors were caused by malice. Unfortunately, POCL
lacked a culture of self-examination. The managers literally did not want to
know about any problems. Secrecy (hiding behind the Official Secrets Act) and
cover-up were the orders of the day.
46.Errors with Horizon were not software problems only, but the result of a
cocktail of problems as outlined above.
Statement of Truth
I believe the content of this statement to be true.
Signed: _:
Page 10 of 11
WITNO9570100
WITNO9570100
Index to First Witness Statement of RUPERT LLOYD THOMAS
No. I URN Document Description I Control Number
1 I WITNO9570101 Timeline of Events by Rupert
Lloyd Thomas dated 22 May
1998
Page 11 of 11