WITNO8110100
WITNO8110100
Witness Name: Mike Croshaw
Statement No.: WITN08110100
Dated: 18/03/2023
POST OFFICE HORIZON IT INQUIRY
FIRST STATEMENT OF MIKE CROSHAW
I, Mike Croshaw, will say as follows...
INTRODUCTION
1. I am a former employee of Fujitsu.
2. This witness statement is made to assist the Post Office Horizon IT Inquiry
(the “Inquiry”) with the matters set out in the Rule 9 Request dated 17
January 2023 (the “Request’).
Page 1 of 8
WITNO8110100
WITNO8110100
BACKGROUND
My professional background is an IT Support Engineer of 20 years
experience.
My role in relation to the Horizon Project was as a third line support engineer
working in the SSC at Fujitsu. My job title was Product Specialist.
KEL system
5.
The KEL database was simply a knowledge database for use of the SSC in
diagnosing issues. Every time we found an issue or some useful information
on how something within the system worked we would document it.
The KEL system worked very well for its intended purpose. It was a very
useful knowledge repository.
To search the KEL system the person just had to go to a search page in input
their search terms (i.e. an error message). This would bring up a list of
possible KELs in order of relevance.
PinICL and PEAK systems
8.
I don’t remember PinICL, I either didn’t use it much, or it was phased out
when I joined. PEAK was our in-house call logging system, it was designed
for use by the SSC but over the years was used by other teams. I always
found it very good compared to other systems I used before and have used
since.
Service tickets were assigned by whomever was on triage duty (called Pre-
Scan in the SSC). They were assigned on the basis of skillset and workload.
10.A service ticket would be closed when the issue was considered resolved.
Page 2 of 8
WITNO8110100
WITNO8110100
ssc
11.I was a member of the SSC for two periods of time: (i) August 1998 until
August 2000; and (ii) January 2004 — April 2020. My recollections are very
positive. It was a group of very competent technical people, well managed,
just doing their job resolving issues to the best of their ability. There was
empathy and sympathy towards postmasters when they had problems with
the system and most of the people in the SSC went out of their way to help
the postmasters. Quite a few would work late on a Wednesday for example
because that was the day postmasters had to do their weekly reports. This
often caused them to overrun and people in the SSC would stay on until 7 or
8pm to help them.
12. The SSC provided 3" line support to the Horizon system. This meant it was
the last and most technical line of support before issues would need to go to
the development team. Before tickets came to the SSC they would go
through 1% and 2" line support which were the main customer facing
helpdesks.
13. There were around 30 people on average in the SSC when I was there.
14.Within the SSC there was the manager, and then three or four senior
technicians. These did some management duties but also were the most
experienced technical people. Many years later these positions became
formalised as team leader positions.
15.To get a job in the SSC you had to take a technical test and pass an
interview. There were no formal qualifications required as such but you had
Page 3 of 8
WITNO8110100
WITNO8110100
to prove you had technical experience and ability. As this was a 3 line
support role this had to be fairly considerable.
16. There was a generous training budget for training courses and many people
went on these. We also had regular knowledge transfer sessions from
development whenever there was a new release. Senior technicians within
the SSC would train the more junior people. New joiners were not allowed on
the live system for some months until they had been trained.
17.We were assigned tickets in PEAK. If we didn’t know what to do we would
seek out one of the senior technicians for guidance. For anything that could
impact the live system a second person was required to observe your work.
18. The relationship between development and the SSC was very good. Between
the other lines of support it was less good due to the fact they were located
elsewhere, and had a high turnover rate. But it was still a good relationship
and members of the SSC went up to Stevenage (where 2" line was based) to
do training and improve the relationship. Some members of 2"4 line support
came to work in the SSC for periods of time to gain experience.
19.The call filtration rate from the SMC varied over the years, depending on who
was in the SMC. Generally it was ok and the SSC could cope very well with
the call volume we had.
Identification and rectification of bugs, errors or defects
20.1 have been asked to set out my recollection of the identification and
rectification of the bugs identified in Appendix 2 of Bates and others v. Post
Office Limited (NO. 6). I have no recollection of any of those bugs. It’s been
20 years since I worked on that part of the system and even then it was not
Page 4 of 8
WITNO8110100
WITNO8110100
my main area of support and I did not work much on counter issues. After the
first year in the SSC I don’t think I worked on any counter/reconciliation issues
(but I can’t remember).
21. I don’t remember the specifics of any of the above issues.
22.1 have been asked a number of questions about the process Fujitsu would
follow when a bug, errors or defect was identified:
a. I have been asked who was told of the identification of the bug. That
would depend on the severity of the bug and who was likely to
impacted. Any serious issues were always communicated to POA, to
the best of my knowledge.
b. I have been asked which teams would be involved in developing the
fix. Again that would depend on where the bug was. But normally one
of the development teams would be involved.
c. I have been asked if the fix required changes to Horizon IT System's
code, how this would be implemented. Any fixes would go through a
full QA process, go through the test environments and then be signed
off by POA before going live.
23. I have been asked about in which circumstances a work around would be
offered to an identified bug. I was not involved in those sorts of decisions.
24.1 have been asked if there were any practice, policies or procedures in place
regarding what information could be passed to SPMs in relation to bugs within
the Horizon IT system. I had no involvement at that level. I would update the
PEAK and escalate any serious issues to the SSC manager or one of the
technical leads.
Page 5 of 8
WITNO8110100
WITNO8110100
25.I have been asked if any pressure was applied on me or colleagues not to
pass information to SPMS in relation to potential bugs. Not that I can
remember.
26.1 have been asked if there were any practice, policies of procedures regarding
what information could be passed to Post Office in relation to potential bugs
within the Horizon IT System. Presumably. However, my updates simply got
put into PEAK, I did not communicate directly with Post Office.
27.1 have been asked if I had sufficient time and resources to identify potential
problems within the Horizon IT System. Yes.
28.1 have been asked how the SSC investigated problems that were suspected
to be related to third party software. We looked at whatever information we
had available in the error logs. If we suspected a code level issue that we
didn’t have access to then we would raise a ticket with the vendor in question.
29.1 have been asked if I ever felt under pressure to avoid finding bugs in the
Horizon IT software. No, not at all, quite the opposite.
30.1 have been asked to set out whether any pressure was placed on me or my
colleagues to pass information about bugs to Post Office. Not that I recall.
Remote Access
31.1 have been asked several questions about remote access as a member of
the SSC.
a. I have been asked when remote access would be used for what
purpose. Normally this would be used to access counter logs. It was
heavily regulated and logged. Also to get onto servers on the live
estate if they were having issues, or to retrieve log files. The live
Page 6 of 8
WITNO8110100
WITNO8110100
estate was remote from us, so we had to use remote access for most
call investigations.
b. I have been asked which workstreams or servers we would remotely
access. Any of the host systems on the live estate, correspondence
servers quite commonly in the Riposte era. We would access post
office counters to retrieve logs and look at registry settings.
c. I haven asked whether we accessed the correspondence servers
and/or counter workstations. Yes.
d. I have been asked which tools I would use to access those
workstations/servers remotely. I can’t remember.
e. I have been asked what security measures were in place to prevent un
authorised access. I can’t remember the details, but tokenised security
was in place, it was not easy to access the live estate.
f. I have been asked what audit data was captured and maintained
regarding the use of remote access. I can’t remember the details, but
the system was heavily audited.
g. I have been asked what data could be added, edited, deleted or
otherwise changed remotely. I can’t recall any real details, but we
could change records in the back end databases. These sorts of
changes were heavily regulated and signed off by POA.
General
32.1 have been asked if SPMS had access to adequate training and assistance
on using the Horizon IT System. I think the training they were given could
have been much better, so no.
Page 7 of 8
WITNO8110100
WITNO8110100
33.1 have been asked what I think the cause of the problems experienced in
using the Horizon IT System. Inadequate training was probably the biggest
problem.
34.1 have been asked what changes to the support services would have
improved the advice and assistance received by SPMs. I can’t think of
anything in particular. The SSC was a very well run support unit.
Statement of Truth
I believe the content of this statement to be true.
“GRO
Dated:_ 18/03/2023
Page 8 of 8