WITNO7540100
WITNO7540100
Witness Name: Sam Hardwick
Statement No.: WITN07540100
Exhibit: WITNO754_01/1
Dated: 6 February 2023
POST OFFICE HORIZON IT INQUIRY
FIRST STATEMENT OF SAM HARDWICK
I, Sam Hardwick, will say as follows...
INTRODUCTION
1.
This statement responds to the Rule 9 request I received on 3 January 2023.
This statement is dated 6 February 2023. The statement sets out relevant
information from my time working at the Network Business Support Centre
(‘NBSC’), which was set up to support Post Offices with the rollout of the
Horizon IT system.
The last role I held that was involved with the Horizon rollout ended in 2004.
This was nearly 19 years ago. My last role with Post Office Ltd (“POL”) was in
2005. I haven't retained any documents from this period. The information in
Page 1 of 12
WITNO7540100
WITNO7540100
this statement is thus provided to the best of my recollection, but because so
much time has passed I cannot guarantee its accuracy.
BACKGROUND
. I joined POL (then, as I recall, “Post Office Counters Ltd”) in March 2000. I
had finished my university degree course the previous summer and my job
with the Post Office was my first permanent job. I joined as a Service Support
Advisor. The job was to provide second-line telephone support to Post
Offices, working for the Network Business Support Centre. The NBSC
provided telephone support for all aspects of day-to-day work in Post Offices,
but focused particularly on supporting the then-ongoing rollout of the Horizon
IT system.
. In mid-2001 I was promoted to Incident Analyst, still within the NBSC. My job
was to analyse the performance of the Service Support Advisors and the
second-line NBSC operation, and to make recommendations about how we
could improve the service to Post Offices.
. In 2003 I was promoted to Problem Manager. In this role I took on mini-
projects to improve aspects of POL’s operations that could not be solved by
the NBSC advisors as individual cases, or where a recurrence of a particular
issue or case-type pointed to the need for a more comprehensive solution.
The NBSC largely followed an ITIL support framework and adopted some of
its terminology, including the distinction between “Incidents”, which were
Page 2 of 12
WITNO7540100
WITNO7540100
individual support cases, and “Problems”, which were the next order of
magnitude and could be one serious case or a trend of smaller incidents.
. In 2004 I left the NBSC to work in POL’s product support function, and my
involvement with Horizon-related issues ceased.
. Before I began my first role, as Service Support Advisor, I attended a training
course in Sheffield with other newly-recruited advisors to learn how to use the
Horizon system. I believe the training lasted two weeks. A POL trainer taught
us how the system worked, up to and including how to complete a cash
account balance, which referred to the regular reconciliation process that Post
Offices undertook. To do this, every new advisor had their own assigned
Horizon terminal, set up with a dummy Post Office, dummy transactions and
balance entries. We went through balances and learned how to investigate
balancing discrepancies.
. After completing this first round of training, I began work in the NBSC itself,
near to Wath-upon-Dearne. Before handling queries from Post Offices directly,
I had to listen in to more experienced advisors and attend more training
sessions. These included more advanced Horizon processes, including how
Suspense Accounts worked, and general training on NBSC processes and
procedures.
. The NBSC provided several layers of support for Post Offices in their use of
the Horizon system. The first two layers were known as “Tier 1” advisors and
Page 3 of 12
WITNO7540100
WITNO7540100
second-line “Tier 2” advisors. Tier 1 advisors were the first point of contact
and could resolve simple queries based on a database of information that
they could access quickly. The database was known as the “Knowledge
Base”. If they couldn’t resolve the query, it could be referred to “Tier 2”. This
meant referral to a Service Support Advisor, which was my job.
10.Service Support Advisors, though they were measured on speed-of-resolution,
among other things, were also able to pause and investigate their “incidents”,
seeking assistance from colleagues as they needed, both within NBSC and,
less commonly, elsewhere in POL. We also had extra referral options. For
problems that indicated a technical issue - for example the failure of a piece of
Horizon equipment - we could refer the Post Office to the Horizon System
Helpdesk (“HSH”). This helpdesk was managed by Fujitsu. Service Support
Advisors could also contact regional managers to seek support for the Post
Office, or Product Support staff for more complex product questions, or the
Problem Management team within NBSC for information about ongoing issues
affecting Post Offices. There was also a complaints team, known as “Customer
Relations”, who would manage cases in which Post Offices were unhappy with
the service they received.
11.My impression from the outset was that the NBSC was a comprehensive,
centralised support service for Post Offices. “Tier 1” advisors were often
recruited from other, nearby contact centre operations and relied on access to
information rather than their own knowledge and training. But from “Tier 2” and
above, staff were well-trained and surrounded by an extensive support
Page 4 of 12
WITNO7540100
WITNO7540100
structure. Morale was generally good and turnover was low. There was a
collective will to offer a good service to Post Offices and, with occasional
exceptions brought about by difficult interactions, staff had a positive
impression of the work staff in Post Offices did. As with all my answers,
however, it is important to note that these impressions apply to my time with
the NBSC between 2000 and 2004. I can’t say anything about the period that
followed, but I do recall a sense that the likelihood of budget cuts, which began
towards the end of my time there, might result in a poorer service.
12.In all of my roles with the NBSC - but especially when I was handling calls
from Post Offices directly, as a Service Support Advisor - I was conscious of
the complexity and difficulty inherent in running a Post Office. Staff often
commented on the vast and growing range of services provided at the Post
Office counter and how difficult it must be to try to manage all of them,
especially when that was combined with running a local shop, as it was for
many Subpostmasters.
13. This complexity was clear in many interactions with Post Office staff, but most
obviously when handling cash account balance queries. This is where our
support for the Horizon rollout was at its most prominent. Staff, most
commonly Subpostmasters, would contact the NBSC because their balance
had produced a discrepancy and they did not understand why. Sometimes
callers were distressed because the implication was that they might have to
bear a loss (though unknown surpluses were also possible, and were not
always less problematic). Service Support Advisors could usually find the
Page 5 of 12
WITNO7540100
WITNO7540100
source of the problem, sometimes on the call itself and sometimes after
consultation and investigation. I became quite experienced with calls like
these. They remain to this day one of the most satisfying things I’ve done in
my career. The combination of problem-solving and the sense of having made
a material difference was rewarding. I know many other advisors felt this way.
14.1 recall that there were times, but only rarely, when we couldn't solve cash
account discrepancies quickly. In these situations we would hold the case (the
“incident”) open and coordinate supporting activity. This could include
contacting the regional manager (known as RNMs, “Retail Network
Managers”, then as RLMs or AMs, “Retail Line Managers” or “Area
Managers”) and seeking a visit to investigate the problem in person. It could
mean contacting the national reconciliation team in Chesterfield, known as TP,
“Transaction Processing”, to check if any error notices were outstanding that
would issue corrections to the balance. Or it might mean seeking a technical
investigation through the Horizon System Helpdesk or our own Problem
Management team to establish the cause of the problem. Sometimes these
investigations led to visits from Fujitsu engineers. While things remained
unresolved, Post Offices moved their discrepancies to Suspense Accounts so
that they could continue daily and weekly operations while the cause and
corrective action were established.
15. Throughout this time, I don’t recall ever being aware of an ongoing, known-
but-unsolved defect within the Horizon system that would create cash account
discrepancies. The nearest I came to this was incidents of the type I refer to in
Page 6 of 12
WITNO7540100
WITNO7540100
paragraph 14, in which NBSC staff couldn't find an explanation and so the
case was referred to the Horizon System Helpdesk. Our role in such cases
was to make sure we’d made whatever checks we could with the Post Office,
and so to make sure we had a strong case for a technical investigation and
avoid unnecessary referrals. I do not recall whether we were always told
about the subsequent resolution in these cases or whether incidents were
handed off entirely. I do recall feeling confident that, having done what we
could, we were referring incidents to ‘the right place’, where technical
investigation would accurately determine the cause of the problem, albeit with
some chasing up required periodically. Moreover, we expected that
information about any recurring problems and associated workarounds or
fixes would be shared with us so that we could advise Post Offices
accordingly. In my later role as Incident Analyst at the NBSC, it was part of my
job to make sure all our advisors had access to the latest information. We did
this with what we called a ‘bulletin board’ - essentially an internal web page
that we could update with changing information and advice.
16.1 want to be clear that I do not question what we now know about Horizon and
the defects that it had. In this statement I am speaking only about the
perceptions I recall holding at the time. With hindsight it is possible, perhaps
likely, that there was a collective overconfidence in the Horizon system at the
NBSC. It did not feel that way to me then. I think it fair to say that we knew
things could go wrong with Horizon - we expected that given the scale and
complexity of the rollout, and in many ways that was why we were there - but I
felt we had broadly the right tools and escalation points to provide support.
Page 7 of 12
WITNO7540100
WITNO7540100
17.We were aware through our work and sometimes through the media that legal
action related to discrepancies was sometimes taken against Subpostmasters.
In the roles in which I worked at the NBSC, I don’t recall being aware of the
details of these cases, only that they existed in what seemed numbers small
enough to ascribe them in our minds to rare individual circumstances, not to
systemic problems. We believed that thorough investigation, extending
beyond our roles and including some or all of the activities I describe in
paragraph 14, would have taken place before any legal action was taken,
though we were not involved in it. It was thus difficult for us then - and even
more difficult for me now - to judge any case individually. I can say, however,
that my own work at the NBSC gave me confidence generally in the Horizon
system’s accounting capability. And I recall feeling that where things went
wrong, either the NBSC or, failing that, Fujitsu, would ultimately be able to
determine the cause accurately and make the right corrections.
18.Given what we know about Horizon’s defects, and their tragic consequences,
the question of whether the NBSC provided adequate support to
Subpostmasters is a difficult one. Perhaps the existence of the scandal itself
proves that it did not. While I worked there, my answer to this question would
have been ‘yes’, though I might have suggested reducing some of what I saw
as unnecessary bureaucracy and complexity in the operation, driven in my
opinion then by excessive adherence to ITIL support structure.
Page 8 of 12
WITNO7540100
WITNO7540100
19.With hindsight I am less convinced that a single, heavily-centralised operation
like the NBSC could have adequately supported nearly 20,000 separate Post
Offices through this magnitude of IT rollout, given the complexity of their work
and the range of services they offered. I recall a prevailing sense that
increased centralisation and automation represented the future of the Post
Office. I don’t think this was at all unique to the Post Office; it was prevalent in
organisational thinking at the time. But for us it meant that in-person field
support for Post Offices, such as RNMs, then RLMs and AMs, was considered
a last resort. Colleagues in those roles were usually stretched, and thus were
to be used as sparingly as possible. As an NBSC advisor I felt that recourse to
field support to help a Post Office was in effect a failure on my own part to
resolve the situation, and I took pride in not needing it often. This was a
personal feeling and advisors were not directly incentivised to avoid using
field support, nor explicitly discouraged from doing so. But the mechanics of
the NBSC environment, in which the quality of its service was closely
associated with its capacity to resolve queries and problems efficiently,
implicitly discouraged processes that necessarily meant slower or deferred
resolutions. I now think it possible that greater appreciation for - and
investment in - in-person support would have led to better outcomes.
20.In this statement I have concentrated on my time as a Service Support
Advisor. My experience in my subsequent roles with the NBSC, as Incident
Analyst and Problem Manager, was consistent with these recollections. I
focus on the first role because it involved frequent, direct contact with Post
Page 9 of 12
2
=
22.
WITNO7540100
WITNO7540100
Offices, and mainly Subpostmasters, so it provided first-hand experience of
the support the NBSC provided.
.I was asked subsequently by the inquiry - on 26 January 2023 - about my time
working as a Problem Manager and specifically about a Fujitsu document,
URN POL00001101 (Exhibit WITNO754_01/1), in which an incident is
described involving a reported discrepancy from a Post Office branch. The
document indicates I was Problem Manager for this incident, at least for a
period. I don’t recall the case specifically but I have no reason to doubt the
document is accurate in this regard.
As Problem Manager for an incident like this, in which NBSC could not identify
the cause of the error, my job would have been as a coordination point
between NBSC and Fujitsu. Often this meant ‘making the case’ on behalf of
the Post Office involved so that the right technical investigation took place.
Some of the updates in POL00001101 show me taking this role. In these
cases generally, I would make sure the NBSC provided information required
by Fujitsu and would pass on status updates from Fujitsu to them, keeping
NBSC colleagues and in turn the Post Office informed of developments. We
would have expected Fujitsu to investigate, with direct access to technical
data, logs, etc, that at NBSC we couldn't see, and establish the cause of the
problem, be that an error on the branch’s part or something connected to
Horizon itself. I don’t recall ever meeting Fujitsu in person to talk about the
incident in POLO0001101 or others like it. I wouldn't rule that out entirely, but I
think it more likely that communication would have been by telephone or
Page 10 of 12
WITNO7540100
WITNO7540100
email. It is possible too that I had moved to my subsequent role before a
conclusion was reached in this case. One characteristic I recall when working
as a Problem Manager is that incidents would sometimes change ownership
within our team, either because a colleague moved to a new role, or because
areas of responsibility were reshuffled. Longstanding incidents could therefore
have several owners between their opening and closing.
Statement of Truth
I believe the content of this statement to be true.
Dated: 6 February 2023
Page 11 of 12
Index to First Witness Statement of Mr. Sam Hardwick WITN0754
WITNO7540100
WITNO7540100
Document Control
No. I URN Description Number
PEAK Incident Report
1 POL00001101
FAD115026
VIS00002115
Page 12 of 12