9 November 2022 – John Simpkins and Mark Ascott
Hide video Show video
(9.57 am)
Mr Beer: Good morning, sir, can you see and hear me?
Sir Wyn Williams: Yes, thank you.
Mr Beer: Thank you, may the witness John Simpkins be sworn, please?
Sir Wyn Williams: Yes, of course.
John Simpkins
JOHN SIMPKINS (affirmed).
Questioned by Mr Beer
Mr Beer: Good morning, Mr Simpkins. My name is Jason Beer and I ask questions on behalf of the Inquiry.
In front of you, there should be a witness statement in your name.
John Simpkins: Yes.
Mr Beer: Can you turn, please, to the last page of it. It is 18 pages in length, dated 4 August 2022, and for the transcript the reference is WITN04110100.
On page 18 is that your signature?
John Simpkins: Page 19, yes.
Mr Beer: It’s page 19, is it? You are quite right. Are the contents of it true to the best of your knowledge and belief?
John Simpkins: They are.
Mr Beer: A copy of that will be uploaded to the Inquiry’s website. I’m not going to ask you about every part of it, just selected parts. Do you understand?
John Simpkins: I do.
Mr Beer: Can you tell us your qualifications, please?
John Simpkins: I studied software engineering at University of Birmingham. I am a member of the British Computer Society, Chartered IT Professional and an Incorporated Engineer.
Mr Beer: You joined Pathway, ICL Pathway Limited, in 1996, July 1996; is that right?
John Simpkins: Yes.
Mr Beer: As an application developer; is that right?
John Simpkins: Correct.
Mr Beer: Can we look at paragraph 9 of your witness statement please, which is page 3. Just wait a moment, it will come up on the screen. You say in paragraph 9:
“While I was initially taken on as an Application Developer, I only remained in this role for a very short time and did not in fact develop any aspects of the Horizon system myself. During my time as an Application Developer, I worked with Dai Jones to learn the coding language being used at the time.”
When you were working with Dai Jones, was there any discussion about the quality of the coding language being used at the time?
John Simpkins: No, I was really only training at that time, so I was being taught how to interact with the Riposte system.
Mr Beer: Did Dai Jones ever discuss with you the quality of the Coding on the Escher product known as Riposte?
John Simpkins: Not at that time.
Mr Beer: I said at any time.
John Simpkins: Yes … not unless there was a PinICL that was raised on it.
Mr Beer: I’m sorry?
John Simpkins: Not unless there was any calls raised on the code.
Mr Beer: Can you recall whether there were?
John Simpkins: There were many calls raised on the code over the years. I don’t know whether Dai Jones raised any of those calls.
Mr Beer: Can you remember any wider discussion with Dai Jones about the quality of the coding on the Escher product Riposte?
John Simpkins: No, I don’t.
Mr Beer: You don’t?
John Simpkins: I don’t recall any further conversations with Dai Jones. I only worked with him for a very short amount of time.
Mr Beer: After that initial period of training as an application developer, when you worked with Dai Jones, did you have cause to work with him again?
John Simpkins: No.
Mr Beer: You say the language was known as Visual Basic:
“A key role of the development team was to ensure that the Visual Basic coding being used by the time interfaced properly with Escher’s software product, Riposte …”
Did it interface properly with Escher’s product, Riposte?
John Simpkins: Yes, that was the way it was – that was the way we interfaced with that product, that was –
Mr Beer: I know it was the way. I was asking you did it interface properly?
John Simpkins: Yes.
Mr Beer: There were no problems with it at all?
John Simpkins: With Riposte or Visual Basic or the interaction? There was … I don’t recall any problems with the Visual Basic and the interaction with the DLs(?) between Visual Basic and the Riposte application.
Mr Beer: Then over the page, you say:
“Access to the Escher source code was only granted to the development team if absolutely necessary.”
So, to your understanding, it wasn’t a question of intellectual property rights preventing any access to the Escher source code; is that right?
John Simpkins: I believe so. I think we had a copy of the source code on the sixth floor in a safe in case it was ever required, but I don’t recall it ever being used.
Mr Beer: You say that access was only granted if absolutely necessary. Was it necessary?
John Simpkins: I don’t recall it ever being used.
Mr Beer: But that facility was there?
John Simpkins: That facility was there, yes.
Mr Beer: It wasn’t that you could never have access to it?
John Simpkins: I believe the reason it was there was so that people could have access to it.
Mr Beer: Yes, thank you. You then moved into the software support centre. Is that called the SSC?
John Simpkins: Correct. It was initially the system support centre and then, I think –
Mr Beer: I was about to ask. The term “software support centre” and “SSC”, is that used interchangeably sometimes with “system support centre”?
John Simpkins: It is. I believe, originally, it was “system support centre” up until after Mik left and I think it got changed to “software support centre” after that time.
Mr Beer: You have remained, I think, in the SSC for 26 years now. You are currently a team leader in the SSC?
John Simpkins: That’s correct.
Mr Beer: Before you became a team leader in the SSC – I think that was in 2010 – what was your job title?
John Simpkins: Project specialist.
Mr Beer: Was that the same for the previous 14 years?
John Simpkins: Yes, I think everyone had that title, really.
Mr Beer: What was the role of a project specialist?
John Simpkins: It was to receive tickets or we sometimes had direct email and we would investigate problems on the live system and then potentially reports, as well, to service management. We try and produce workarounds if there was an issue and try and resolve problems that were passed to us, really, on the live estate.
Mr Beer: Was there a level below a project specialist in the SSC?
John Simpkins: Not by terminology. You had areas of specialism, so there were many products that made up the solution, like the databases and Riposte, the agents and – and people were specialists in certain areas, but I think they were all called project specialists. You might be working on Tivoli, the rollout database, ACMS, or any of these other areas that you were still, I think, called a project specialist.
Mr Beer: In, say, 1999/2000 at the rollout stage of Horizon how many people worked in the SSC?
John Simpkins: I think we went up to about 25.
Mr Beer: At, say, 2010, at rollout stage of Horizon Online, how many people worked in the SSC?
John Simpkins: Probably slightly more. I think Mik was hiring at that time, but yes, I mean – I think it probably topped out around 30, but maybe around 25 to 30.
Mr Beer: How many team leaders were there in the SSC, say, at the first date that I mentioned, 1999/2000?
John Simpkins: There was only a manager at that point, no team leaders.
Mr Beer: When were team leaders introduced?
John Simpkins: 2010.
Mr Beer: So when you became one?
John Simpkins: Yes. So after Mik left – 2009, I think – we had Tony Little step in for a while and then Steve then took over in 2010, and he introduced the three team leaders.
Mr Beer: The Steve you refer to there, is that Steve Parker?
John Simpkins: That’s correct.
Mr Beer: Who did each of the SSC team leaders report to: to him?
John Simpkins: To him.
Mr Beer: He was the SSC manager; is that right?
John Simpkins: Correct.
Mr Beer: Do you know to whom he reported?
John Simpkins: Not entirely – Steve Muchow – I’m not sure when Steve Muchow left. Peter Bird, I’m not sure when Peter Bird left. They were levels above him. I’m not sure, I’m afraid.
Mr Beer: Okay. Can we look at paragraph 7 of your witness statement, please, which is at the foot of page 2. You say in the second sentence:
“The team [that’s the SSC] does not support the hardware or operating systems. The team had a good interaction with the testing teams and development to supply evidence and find possible ways to recreate defects on test equipment. We also interacted with subpostmasters when gathering evidence or providing support. The … SSC was not responsible for reporting to Post Office.”
Who was responsible for reporting to Post Office?
John Simpkins: I know that Mik did do monthly reports.
Mr Beer: And Mik –
John Simpkins: So Mik Peach did monthly reports up to his management. There was also service management –
Mr Beer: Sorry, just stopping there. You say that he, Mik, did reports up to his management?
John Simpkins: Yes.
Mr Beer: Was that still within Fujitsu or ICL?
John Simpkins: Within Fujitsu. He also – I’m not sure of the date totally. He introduced something called the SMP, service management portal, which he –
Mr Beer: Can you explain what the SMP was?
John Simpkins: So it was a website that Mik introduced and wrote and it was for him to put reports on and I believe the change management OCPs were also copied onto there and that was for Post Office to have visibility of these.
Mr Beer: Did Post Office have direct access to the SMP?
John Simpkins: Yes.
Mr Beer: You were about, in your first answer, to go on to speak about the service management team?
John Simpkins: So, yes, service management was really the interface, I believe, between support issues and Post Office.
Mr Beer: Just stopping you there, where were they based?
John Simpkins: They were in – they’re Fujitsu. I think they were in Bracknell, as well. And then I was going to talk about the MSU, the management support unit. They did the reconciliation and they reported –
Mr Beer: The reconciliation of what?
John Simpkins: Sorry, if there were any reconciliation incidents, so they would then report those reconciliation incidents back to the Post Office.
The term I remember currently is BIMs, business incident management, but there is also – reading through PinICLs, some red but I don’t know what red represented.
Mr Beer: You say in paragraph 25 of your witness statement:
“To the extent that there were any known defects when releases were rolled out, my understanding is that this would have been communicated to Post Office, either by the Service Management team … or by other ICL … teams. I was not involved in communications with Post Office in this regard, neither am I aware of how or if such issues were communicated to subpostmasters.”
Later in your statement, in paragraph 47, in relation to the accuracy and integrity of data recorded and processed on the system, you say:
“I cannot comment on how general issues would be relayed to Post Office but, in respect of individual incidents, I believe this information was passed back to the Post Office through the BSU/MSU or Service Management.”
What’s the basis for those understandings and beliefs that you give?
John Simpkins: So the first one was about projects, so when we have new functionality entered into the system, it is normally entered in via project. It is not normal support at that stage and projects have a – projects are managed and, I believe, they are fed back through the project management chain, that –
Mr Beer: Yes, and what was the basis for that belief?
John Simpkins: I have been involved in some projects.
Mr Beer: I’m talking about this one.
(Pause)
Mr Beer: It is paragraph 25, so when known defects – when the releases were rolled out your understanding that this would have been communicated to the Post Office. I’m asking you for the basis for that belief, please?
John Simpkins: Just because projects reported back. Sorry, I’ve got nothing more than that.
Mr Beer: So it’s a general understanding that that’s what should happen –
John Simpkins: Yes.
Mr Beer: – between a service provider, Fujitsu, and its client, the Post Office?
John Simpkins: Correct.
Mr Beer: You haven’t got any actual knowledge of whether that did happen?
John Simpkins: I’ve got no actual knowledge.
Mr Beer: You see, we have heard some evidence in the Inquiry that, because this was a PFI – Public Finance Initiative – framework, under which the services were being provided, the Post Office had what was described as limited or partial visibility of the design approach, the development approach and defects. Were you aware of that or not?
John Simpkins: Not particularly, no.
Mr Beer: In relation to the comment in paragraph 47 where you say “I believe that information”, that’s general issues – sorry, specific individual incidents, you believe that information was passed back through the BSU/MSU or service management.
Again, what’s the belief for that, or the basis for that belief and understanding?
John Simpkins: So if there was an issue that was a new issue, that would be put into the monthly reporting by the SSC manager and service management were involved in resolution of issues. They were the ones who did the reporting. The BSU is, if it’s a reconciliation incident, they would do the reporting.
Mr Beer: Do you know from personal knowledge the extent of the reporting by MSU/BSU?
John Simpkins: I’m sure in the court case there was a – released a monthly service management report. I can’t remember which incident it referred to, but it had broken down about recent issues.
Mr Beer: So the court case you’re referring to is?
John Simpkins: The GLO, sorry.
Mr Beer: What’s your knowledge of the GLO that you’re referring to there? Are you referring to the judgment, or –
John Simpkins: There was some evidence released as part of the GLO and that included a monthly report from the Fujitsu service management team.
Mr Beer: So the “evidence”, whose evidence are you referring to?
John Simpkins: I couldn’t tell you whose evidence it was.
Mr Beer: I’m just trying to explore where you are getting this knowledge from. Is it as a result of –
John Simpkins: Yes, I viewed this document that was part of the released documents part of the GLO.
Mr Beer: Sorry, I’m just going to press you a little further.
John Simpkins: Yes, sure.
Mr Beer: You viewed a document that was included as evidence in the GLO?
John Simpkins: That was released. I was following the GLO case and one of the documents in there was – that was released as part of the evidence was a service management report.
Mr Beer: How were you following the GLO?
John Simpkins: We followed the Twitter feed and also there were some solicitors – we provided some information to the solicitors.
Mr Beer: So the thing you’re telling us about now is based on reading a Tweet about the conduct of the GLO?
John Simpkins: And seeing a document that was from that.
Mr Beer: Sorry, and seeing a document?
John Simpkins: There was a document that was released as evidence which was a service management monthly report from Fujitsu to Post Office.
Mr Beer: Okay. Can I move on to helpline systems, please, and, as the first witness who is giving evidence to the Inquiry about support services available to subpostmasters, I would like to use you, please, just to confirm the various levels of ICL and Fujitsu support that were available.
I think it is right that, initially, there were three levels of support and then that grew to four; is that right?
John Simpkins: Yes.
Mr Beer: Was the first line of support the subpostmasters initial point of contact –
John Simpkins: Yes.
Mr Beer: – and, essentially, Fujitsu’s gateway to the remainder of the service support?
John Simpkins: Yes.
Mr Beer: Was that carried out by the Horizon System Helpdesk which was later known as the Horizon Service Desk?
John Simpkins: Correct.
Mr Beer: Would this be a fair summary: it would seek to resolve basic queries and then pass on those that it couldn’t rectify to the second line of support?
John Simpkins: Yes.
Mr Beer: Initially, did the Horizon System Helpdesk people work in Feltham?
John Simpkins: Yes.
Mr Beer: Was that where you worked –
John Simpkins: Yes.
Mr Beer: – as part of the SSC?
John Simpkins: Yes, Feltham A1.
Mr Beer: I’m sorry?
John Simpkins: Feltham A1. There are multiple Fujitsu buildings in Feltham.
Mr Beer: I think you say in your statement that it was in fact in the same room as you; is that right?
John Simpkins: That’s right. there was a custom built room for AGL, which brought the parties together. So in the same room we had the HSH, we had us, the SSC, the EDSC, we had the operations team and we had GiroBank.
Mr Beer: And how many of them were there, say, at 2009/2008?
John Simpkins: Just a couple.
Mr Beer: Just two?
John Simpkins: Right at the beginning, 1997 – 1996 to 1997, only a couple, very, very limited. When we moved into Bracknell and they moved out, I don’t know how many there were then.
Mr Beer: Did they move to Bracknell?
John Simpkins: Sorry, they moved to Stevenage.
Mr Beer: Wasn’t that the second line of support that moved to Stevenage?
John Simpkins: The second line were also in Stevenage.
Mr Beer: So, just to make it clear, first line of support also moved to Stevenage; is that –
John Simpkins: Correct.
Mr Beer: When was that?
John Simpkins: I’m presuming it was when we also moved out in 1997 but I would have to ask and check.
Mr Beer: The second line of support for software, was that provided by the system management centre, or SMC?
John Simpkins: Correct.
Mr Beer: Would this be a reasonable description of it: it sought to resolve technical problems itself and acted as a gatekeeper and filter to the third line of support?
John Simpkins: Yes.
Mr Beer: It was also involved in identifying system events that could indicate a software problem had arisen?
John Simpkins: Yes.
Mr Beer: There was also, is this right, another second line of support for hardware, as opposed to software?
John Simpkins: Yes. The engineers – I wasn’t very much involved in the engineering. Oh, unless you’re talking about the ops team – no, the hardware would be engineering.
Mr Beer: They initially worked in Feltham, is that right, the system management centre?
John Simpkins: I don’t think they were in place when we were in Feltham.
Mr Beer: Okay. So what, they only ever existed in Stevenage?
John Simpkins: Correct.
Mr Beer: The third line of support, I think – is this right – provided by a variety of teams depending on the issue, the first of them was you, the system service centre or SSC, and that had, as its focus, investigation and rectification of software problems?
John Simpkins: Correct.
Mr Beer: There was the management support team or management support unit, MSU. That monitored and managed reconciliation errors?
John Simpkins: Yes.
Mr Beer: A reference data team, were you aware of them?
John Simpkins: I was. They eventually joined into the SSC.
Mr Beer: Did they focus on errors or problems in or with the reference data upon which Horizon relied?
John Simpkins: Yes.
Mr Beer: Then operational services division, which I think you called operations, they provided support to network and central system incidents?
John Simpkins: Yes, yes. They looked after the data centres, yes.
Mr Beer: Then the fourth line of support involved development teams that would make changes to Horizon coding to resolve identified errors, bugs and defects; would that be right?
John Simpkins: Yes.
Mr Beer: Would you agree that your part of the third line of support, it’s intended purpose and functions were to provide a support service to resolve technical problems in the minimum time possible and the minimum disruption to the service and to the network?
John Simpkins: Yes. When you say “network”, you don’t mean physical network, you mean as in …
Mr Beer: The system.
John Simpkins: Yes.
Mr Beer: To provide a centre of technical expertise for customer services more generally, providing technical advice, guidance and expertise –
John Simpkins: Yes.
Mr Beer: – and to maintain the KEL database?
John Simpkins: Yes, we ran the KEL database.
Mr Beer: Would you agree that the SSC was at the heart of the support services provided for Horizon?
John Simpkins: The software support services, yes.
Mr Beer: In particular, it occupied a central position in the investigation of bugs, errors and defects?
John Simpkins: Yes.
Mr Beer: If you look at page 19 of your witness statement at paragraph – sorry, paragraph 17 of your witness statement, on page 7, about six lines in, you say:
“If first line support could not resolve the issue and it was related to the software, it would be escalated to the second line support team.”
Do you see that sentence?
John Simpkins: Yes.
Mr Beer: Can you assist us, how would someone in the first line of support on the end of the phone know that an issue that was being reported to them by a subpostmaster was or was not related to software?
John Simpkins: I didn’t work in the HSH but I believe they had scripts to follow, which would help them.
Mr Beer: So a postmaster phones up and says “I’ve got this issue, there’s a reconciliation problem”, how would the first line support know that that related to software?
John Simpkins: As I say, I did not do their role. However, I do believe they had scripts to follow which they would ask them to check various things throughout the script.
Mr Beer: I’m going to press you a little bit further because of what you said in your statement.
John Simpkins: Yes.
Mr Beer: Having gone through the script, how would the first line support know that the issue related to software and therefore pass it to the second line?
John Simpkins: I presume that they would get to the end of the script and it hasn’t resolved the issue and then they would pass to the second line team.
Mr Beer: So it must relate to the software?
John Simpkins: It must not always relate to the software but, because the script will only test so many things –
Mr Beer: What training did the first line support have to make decisions about whether an issue related to software or did not?
John Simpkins: I couldn’t tell you what the training of the first line was.
Mr Beer: Were, to your knowledge, subpostmasters told that there were three and then four possible lines of support?
John Simpkins: I don’t know what the subpostmasters were told about the support hierarchy.
Mr Beer: You don’t know what they knew?
John Simpkins: I don’t know what the subpostmasters knew. I know that quite often one of them would talk to us, but I don’t know if they knew what role we were providing. I think they would ask for people by name.
Mr Beer: The subpostmasters would?
John Simpkins: Yes. There was definitely some PinICLs were a subpostmaster who has been talking to someone in third line support would ask could they talk to that person again.
Mr Beer: Yes, so they have had some dealings with them –
John Simpkins: Correct.
Mr Beer: – they would say “Can I speak to John again please?”
John Simpkins: Exactly.
Mr Beer: But they wouldn’t know when they’re phoning up “I’ve got a problem with software, I need to speak to John”?
John Simpkins: No, no idea.
Mr Beer: With what frequency would software issues, to your knowledge, be referred to second line support?
John Simpkins: I couldn’t tell you, but I’m sure from the PowerHelp tickets, you could work it out because they’ve got the team transfers in the PowerHelp. I could tell you that about 2 per cent of calls came from PowerHelp to PinICL and about half of those were raised by subpostmasters, so about 1 per cent of calls were raised by subpostmasters to the SSC, but I couldn’t –
Mr Beer: And the other 1 per cent?
John Simpkins: The other one was BSU reconciliation – sorry, issues passing from other teams, not necessarily the subpostmasters, but SMC or BSU.
Mr Beer: Why were the teams split up?
John Simpkins: Why were the HSH and SMC split up or?
Mr Beer: Yes.
John Simpkins: I presume that the SMC –
Mr Beer: Don’t worry about presumptions or speculation; do you know?
John Simpkins: I don’t know.
Mr Beer: If you don’t know an answer to a question it’s best to say it –
John Simpkins: Okay.
Mr Beer: – rather than put together maybe fragments of evidence and to speculate.
John Simpkins: Okay.
Mr Beer: Were you party to any discussion over whether the support teams should remain together, rather than splitting up into different offices?
John Simpkins: No.
Mr Beer: Was there, within third line support, ever discussion over trends or patterns that emerge from the nature of calls that were being received, for example a theme is emerging that there are constant problems with balancing?
John Simpkins: Definitely would look at trends and investigate things. If you never got quite to the bottom of something, you saw something again, you would continue. You would normally raise a KEL on a topic, and then you would say on there, you know, “If this happens again could you please examine this and this”. Sometimes evidence was too old by the time we got there.
Mr Beer: What do you mean by that “sometimes the evidence was too old”?
John Simpkins: Sometimes the evidence had been archived away.
Mr Beer: Archived by who?
John Simpkins: By Riposte.
Mr Beer: What difficulty did that present?
John Simpkins: It meant that you could sometimes not get to the bottom of an issue so you would raise a KEL and, if it occurs again, then you know where to look at straight away.
Mr Beer: When you say it had been archived away by Riposte, was that a function of Riposte that could not be broken into or interfered with?
John Simpkins: Archiving definitely could be changed, yes, and, actually, there were features to turn archiving off if, for example, the system had been off for a long time but, yes, archiving could be changed.
Mr Beer: That’s a separate issue, whether archiving could be changed. In respect of data that had been archived, was it impossible to look at it?
John Simpkins: It wasn’t impossible because it would have gone to audit, but – yes, so you could have got information from audit.
Mr Beer: You said that it was difficult sometimes because Riposte had archived the material. Did you ever – or were you ever a part of a process to obtain material from archive, in order properly to investigate an issue?
John Simpkins: We definitely made a request to the archive team, yes.
Mr Beer: So that was a theoretical difficulty rather than an actual one; would that be right?
John Simpkins: Yes. Sorry, I was trying to come up with reasons why you may not have got to the bottom of a problem.
Mr Beer: Yes, and why were you trying to come up with reasons why you might not have got to the bottom of a problem?
John Simpkins: Because you were asking about how you may – the process for going around to documenting a trend.
Mr Beer: Yes, and so this is a theoretical obstacle that could be overcome?
John Simpkins: That one was.
Mr Beer: If you wanted to get to the bottom?
John Simpkins: Yes.
Mr Beer: What other obstacles would there be in getting to the bottom of a problem?
(Pause)
John Simpkins: I’m going to have to look at some PinICLs or KELs and come back on that.
Mr Beer: I’m sorry?
John Simpkins: I would look at some PinICLs and KELs and come back to you about reasons why we have raised some to trend analysis, if that’s okay.
Mr Beer: Does it follow from the need to carefully think about it that there’s nothing obvious that strikes you –
John Simpkins: There’s nothing obvious, yes.
Mr Beer: – that prevented, other than the very theoretical thing that you have mentioned, in getting to the bottom of a problem?
John Simpkins: Yes.
Sir Wyn Williams: Mr Beer, could the statement be taken down from my screen?
Mr Beer: I’m so sorry, sir. Yes, of course.
Sir Wyn Williams: Thank you.
Mr Beer: Was the main mechanism for picking up themes the use of the KEL system?
John Simpkins: Not particularly. The KEL system was very useful for SMC with eventing. It was useful to see if this issue had occurred before but, generally, if – things occurred before you tended to know them, so it was a way of say providing advice and guidance on how to deal with something, mainly if you have not seen it very often.
Mr Beer: What was the mechanism, if any, for picking up themes and trends then, if it wasn’t the KEL system?
John Simpkins: The KEL system was good because – sorry, if we had a lot of incidents with the same issues, then if they were actually found to be defects and passed on to fourth line, there would be trends in that because of the number of PinICLs raised and applied to the same products, that you can see in the PinICLs.
If the KEL system was good for identifying if something had occurred before as well, we did sometimes add onto it “Could you add other PinICL references if this reoccurs”, so there was trending in the KEL system as well.
Mr Beer: Was there any other system operated, to your knowledge, to pick up themes and trends in the problems with the system that were being reported to Fujitsu?
John Simpkins: Not in the SSC.
Mr Beer: In any other part of the service help levels of support to your knowledge?
John Simpkins: There were other teams like QFP and –
Mr Beer: What does QFP stand for?
John Simpkins: Sorry, quality filtering process – that would manage incidents to the – so when we passed a ticket in PinICL to the fourth line people, it would often go through the quality filtering process team, who decide where it was to go to, which area of expertise inside the fourth line support teams, and so there was also analysis of when ticket – working out the amount of effort a fix may take, that that was all in part of the development and release process.
Mr Beer: That sounds like it is more about systems control within Fujitsu for the benefit of the efficient operation of the help service within Fujitsu.
John Simpkins: Yes.
Mr Beer: I’m talking about something that’s of benefit perhaps to the Post Office or to subpostmasters, ie something within Fujitsu where repeated errors, bugs or defects, or even repeated calls about the same system issue, for example balancing, were picked up to say “Look, we’ve got a trend developing here, we need to undertake a root cause analysis”, or something like that?
John Simpkins: There was nothing automated that I know of.
Mr Beer: What about people?
John Simpkins: Yes, I mean, there were people in the support teams and –
Mr Beer: Which part of the support teams?
John Simpkins: Sorry, there was nothing in the SSC that I know of that was –
Mr Beer: Had that function?
John Simpkins: – dedicated to do that function. There was customer service and service management teams that –
Mr Beer: What level of the four were they?
John Simpkins: They weren’t support teams, sorry, they were the people that I said would report to Post Office the major incidents, and things like that.
Mr Beer: How would they get to know about any trends or themes that were developing?
John Simpkins: Only if they would be reported up so –
Mr Beer: By?
John Simpkins: By, I would say, the helpdesk, or the SMC, or us, the SSC, through management.
Mr Beer: Did you do that? Did you take a step back? Rather than dealing with the next ticket on the line, did anyone in your team take a step back and say “There’s a theme developing here, there’s an underlying issue, we need to make a reference”?
John Simpkins: I can’t give you any examples of that.
Mr Beer: Can I turn to the Riposte product, please. At page 15, paragraph 48 of your statement, at the foot of the page, you say:
“In terms of deficiencies during this time, there were a number of difficulties arising from the Riposte product. These included malformed messages … and replication issues.”
What were the difficulties arising from the Riposte product?
John Simpkins: So the malformed messages is when a message is missing attributes, so Mr Cipione broke down what a message attribute – Riposte message looks like, and it has different attributes in it, and we used to use a system called a TIP repair tool when these messages were harvested into the TPS system, and some of these attributes were missing. Then we would have to go and look and see where – what was happening on the counter when that message was written to identify what the missing attributes were.
Mr Beer: What was the cause of the malformed messages?
John Simpkins: I don’t know what the underlying root cause of that problem was.
Mr Beer: Was that ever investigated?
John Simpkins: I’m sure it was.
Mr Beer: By who?
John Simpkins: It would have been fourth line support talking to Escher.
Mr Beer: Was the cause of the difficulties the coding?
John Simpkins: I don’t know what the root cause was.
Mr Beer: Were you ever told back down the line what the root cause was?
John Simpkins: Sometimes – if you had a ticket and it was being investigated by fourth line support, you would hold on to a ticket to find out what the root cause was.
Mr Beer: You tell us in your statement that malformed messages could potentially result in a receipts and payments mismatch but this would unlikely have caused the discrepancy, ie a loss or a gain. How would a receipt – a mismatch problem or issue, manifest itself to the subpostmaster?
John Simpkins: They were informed by a message saying that there had been a receipts and payments mismatch and it would be when they produced the cash account, the final cash account, I believe.
Mr Beer: How would the malformed message sometimes cause the discrepancy then?
John Simpkins: The discrepancy – it could affect the primary mappings, so the –
Mr Beer: Sorry, the primary?
John Simpkins: Primary mappings, sorry.
Mr Beer: Can you explain what that is, please?
John Simpkins: So each transaction is added into the cash account using primary mappings. It’s like a tree and it builds up and searches for all those transactions that meet that primary mapping, and they are added together to complete that node, and it is all added up together and, if that primary mapping was missing or malformed, then it wouldn’t get put into the right place as it builds up the cash account.
Mr Beer: To your knowledge, was the root cause of those problems fixed?
John Simpkins: I don’t know.
Mr Beer: Do you know what subpostmasters were told when it was suspected that there was a discrepancy caused by a malformed message?
John Simpkins: They would have had the message on screen saying there was a receipts and payments mismatch and then it would have been investigated. There was an event written, I believe, as well, so – and also harvesting at the TPS database would identify it. So they would – they could raise a call but, also, we would get the ticket from the MSU/BSU.
Mr Beer: I’m talking about what the subpostmaster was told themselves, “Look, there’s a discrepancy, you’ve got this message” –
John Simpkins: I don’t know what they were –
Mr Beer: – “don’t worry, it’s not you, you haven’t done anything wrong, we believe it’s caused by a malformed message”?
John Simpkins: I don’t know what the subpostmasters were told.
Mr Beer: You refer in paragraph 51 of your statement to the fact that:
“There could be many root causes for replication failures between counters. This could include network cable faults, hub faults for large branches, hardware faults and issues with Riposte.”
Can you expand on which of those potential faults were, in your experience, real faults that actually happened in practice?
John Simpkins: I think they all happened in practice.
Mr Beer: Again, to your knowledge, what were subpostmasters told about this? They get the message that you have spoken about saying that there is a discrepancy, a mismatch; what were they told about the cause of the mismatch if it was attributable to one of these things?
John Simpkins: The replication is different to the corrupt primary –
Mr Beer: Malformed message, yes.
John Simpkins: Yes, but the replication would normally be presented to postmasters when they were looking at a transaction, or – and then it’s not there, so run a report and it’s missing some transactions because they did them on counter 2 and they ran a report on counter 1.
Mr Beer: Again, can you help us with what they were told about those?
John Simpkins: Again, no, I can’t tell you.
Mr Beer: Is that because it was somebody else’s responsibility to tell them?
John Simpkins: It would have come in from the HSH.
Mr Beer: You said it would have come in from the –
John Simpkins: When they contact the HSH to report the issue.
Mr Beer: But they don’t know, the subpostmaster, whether this was a hardware fault, they don’t know whether it’s an issue with Riposte, they don’t know whether any of the range of things that you mentioned is a cause of the replication error; all they know is the error message that they’re getting. So what process was there to feedback to them, “Look, you haven’t done anything wrong, you haven’t stolen thousands of pounds here, it’s a problem with our system”?
John Simpkins: So if the – if it was the Riposte one then it wrote an event which was picked up by the SMC and they raised a call and they were contacted – they contacted the subpostmasters for those.
If it was the hardware ones, I don’t know. But, again, that wouldn’t have caused the receipts and payments mismatch.
Mr Beer: Sorry?
John Simpkins: Again, it was about replication, not corrupted notes.
Mr Beer: Yes, for the subpostmaster it may not matter particularly, other than to know that it wasn’t an error of their own.
John Simpkins: Yes.
Mr Beer: But you can’t help us as to who was responsible for feeding that back to subpostmasters?
John Simpkins: I can’t.
Mr Beer: No, thank you.
In paragraph 58 of your statement, you say:
“I am not aware of any practices or procedures that may have been in place to obtain input or feedback from subpostmasters during the pilot and rollout of Horizon.”
Is that because this was a different area of business from you or is it because it didn’t happen?
John Simpkins: I couldn’t tell you because it was a different area from me. If they contacted – if a ticket was raised and came to us, we would talk to the subpostmasters relating to that ticket.
Mr Beer: This is a slightly different issue. This is during pilot and rollout. Were there any problems that were being experienced by subpostmasters, whether there was a mechanism to capture those and to incorporate any fixes to them in the system. You’re not aware of, kind of, that process?
John Simpkins: No, I’m not.
Mr Beer: Can I turn to a different issue then, please. For how long have you known Anne Chambers?
John Simpkins: Quite a long time. She joined the SSC – I can’t tell you how long, but it was many years, more than ten years.
Mr Beer: What was her function in the SSC?
John Simpkins: She was a project specialist. She dealt with counters in particular.
Mr Beer: Was she there from the start, from your recollection?
John Simpkins: Not from the start but she was there a long time.
Mr Beer: How closely did you work with her?
John Simpkins: Very closely.
Mr Beer: Was your contact with her frequent then, on a daily basis?
John Simpkins: Yes.
Mr Beer: How close did you sit from her, physically?
John Simpkins: A couple of desks away. It was a strange arrangement of desks.
Mr Beer: What was her role and function when you worked alongside her?
John Simpkins: She was another SSC product specialist.
Mr Beer: And I think you said specialised in the counters?
John Simpkins: Yes, her area of expertise was in the counters.
Mr Beer: Just explain what specialism in the counters means?
John Simpkins: So when a ticket comes into the SSC, we had a pre-scanner and the pre-scanner’s role was to analyse the ticket, check it had all the information expected on it and then route it to a member of the team in the SSC, based on their workload and their areas of expertise and, as I say, she worked on the counter tickets.
Mr Beer: Did you become aware of her being asked to give evidence in a court case?
John Simpkins: Yes, we were.
Mr Beer: You say “we were”?
John Simpkins: Yes, the SSC as a whole were aware of this.
Mr Beer: Can you remember when that was?
John Simpkins: I can’t remember the exact date, but I do remember that Anne was unhappy to be asked.
Mr Beer: She was unhappy?
John Simpkins: Yes.
Mr Beer: This was before she had actually given evidence; is that right?
John Simpkins: Correct.
Mr Beer: Can you remember whether there was discussion before she gave evidence about her suitability as a witness or the appropriateness of a member of the SSC going along to give evidence?
John Simpkins: I don’t know about that conversation.
Mr Beer: Was there a conversation between you and Anne, or you and other members of the SSC and Anne, about the appropriateness or suitability of her going along to be a witness?
John Simpkins: There was conversations about whether SSC people were the right people to be used.
Mr Beer: Why was there a question over whether SSC people were the right people to be used?
John Simpkins: I think we thought it was more – because we were very – technically specialists in that area and not expert witnesses, we were very unhappy about that process.
Mr Beer: Was Anne Chambers very unhappy about the process?
John Simpkins: I believe she was.
Mr Beer: Did she say that to you?
John Simpkins: I cannot recall the conversation, but I believe she was.
Mr Beer: After she gave evidence, was there any discussion about the appropriateness of her doing so or her suitability as a witness?
John Simpkins: I don’t know if there was anything about her suitability but I know that she fed back to the SSC manager that she didn’t find it at all nice and we – I do not believe that – I believe the SSC manager then pushed back to say – so that it never happened again.
Mr Beer: The SSC manager that she spoke to was?
John Simpkins: Mik Peach.
Mr Beer: You said that the SSC manager, words to the effect of, ensured that it never happened again. Who did Mik Peach take that up with to your knowledge?
John Simpkins: I don’t know.
Mr Beer: What was the issue with her giving evidence then? What was the problem about it?
John Simpkins: We just weren’t expert witnesses. It was a – it did not feel right.
Mr Beer: Do you know why she did it?
John Simpkins: I believe that she was manoeuvred into it. I don’t know if she really wanted to do it. She had dealt with the case, I believe.
Mr Beer: Who was she manoeuvred by?
John Simpkins: I don’t know.
Mr Beer: On what basis do you say that she was manoeuvred?
John Simpkins: I don’t think she would have wanted to do it otherwise.
Mr Beer: Who are the candidates for manoeuvring her into doing it?
John Simpkins: I don’t know.
Mr Beer: Can you help us?
John Simpkins: I would talk to – about the security teams maybe, who would have interfaced with the request for that. I don’t know.
Mr Beer: At what level was her unhappiness at being asked to give evidence and then, after she had done so, expressing her unhappiness about having done so?
John Simpkins: On a scale of –
Mr Beer: Yes, of mildly fed up at the bottom end, to incandescent with rage at the top end, say?
John Simpkins: She was probably in the middle. She was really – said how unpleasant it was and she did not want to do it again.
Mr Beer: For how long have you known Gareth Jenkins?
John Simpkins: Gareth, I think, was there from the beginning. I recall seeing him in Feltham, so it would have been from probably 1996.
Mr Beer: How closely did you work with Mr Jenkins?
John Simpkins: So we interfaced quite a bit about – he was the fourth line and – so the development and architecture, and he was a specialist in the Riposte area, so if we had some issues in that area we would talk to him. He was approachable.
Mr Beer: How frequent was your contact with him?
John Simpkins: Maybe monthly.
Mr Beer: Would that be face-to-face or via emails?
John Simpkins: Normally emails or PinICLs.
Mr Beer: Did you have meetings with him?
John Simpkins: I have definitely been in meetings with him. I think one of my witness ones is meeting with him.
Mr Beer: To your knowledge, what was his function?
John Simpkins: He was either chief technical or he was one of the technical – chief technical people, architects, for the Riposte area and, later on, he was also in HNG-X.
Mr Beer: Were you aware of any discussion about the suitability of him or the appropriateness of him as a witness to give evidence?
John Simpkins: Not until the GLO.
Mr Beer: So after the event –
John Simpkins: Yes.
Mr Beer: – when you saw that issue emerge in the course of the Group Litigation?
John Simpkins: Yes.
Mr Beer: Was there any contemporaneous discussion that you are aware of as to the selection of an appropriate witness to give evidence, either in written form or orally, in criminal proceedings against subpostmasters for theft or false accounting?
John Simpkins: Not that I’m aware of.
Mr Beer: We are aware of an article in Computer Weekly, a trade journal, of 11 May 2009. Can you remember when you first became aware of that?
John Simpkins: In this – I think you mentioned it recently.
Mr Beer: That’s the first you have known of the Computer Weekly article?
John Simpkins: 2009, when I have watched some previous articles in – on the online Computer Weekly about things.
Mr Beer: Does it follow that the Computer Weekly article of May 2009 wasn’t discussed in the office at about the time that it came out?
John Simpkins: I don’t recall that.
Mr Beer: When you say you don’t recall it, that could mean that it may have happened but you may have forgotten, or “I don’t recall it because it is likely that it didn’t happen”?
John Simpkins: I don’t recall it. It could have happened but I do not recall a conversation about it.
Mr Beer: Can I turn to a separate issue, please, the issue of remote access.
Could we look, please, at POL00030029. It will come up on the screen for you.
John Simpkins: Thank you.
Mr Beer: Can we look at page 4, first, please. At the foot of the page this is an email of 13 May 2014, from Sean Hodgkinson. If we just look at the bottom of the next page, please, to see who he was: senior consultant in the audit advisory division of Deloitte, yes?
John Simpkins: Yes.
Mr Beer: Then if we just go back to where we were, please, the previous page. Thank you. You can see that the email of 13 May 2014 is to a range of people. You are not included on this chain but, as we will see in a moment, you end up answering the questions in this chain. Do you remember?
John Simpkins: I do, from reading.
Mr Beer: Yes. I just want to see what the questions were first and this is to a collection of people, I think substantially within the Post Office:
“All,
“Following review of the technical design document in relation to the Branch Database, I had a couple of queries that I was hoping you may be able to help with. If not, please could you direct me toward somebody who may be able to assist:
“1) Balancing Transactions.
“Section 5.6.2 …”
Do you know what that is of?
John Simpkins: No.
Mr Beer: “… describes back end database amendment process which is included by design …”
Then he quotes from the document “Inserting Balancing Transactions”:
“There is a requirement that the SSC will have ability to insert balancing transactions into the persistent objects of the Branch Database. There are reasons for SSC having to do so, eg to rectify erroneous accounting data that may have been logged as a result of a bug in the Counter/BAL.”
Over the page, please:
“SSC will have privileges of only inserting balancing/correcting transactions to relevant tables in the database. SSC will not have any privileges to update or delete records in the database. Any writes by the SSC to BRDB …”
BRDB?
John Simpkins: Branch database.
Mr Beer: “… must be audited. The mechanism for inserting a correction record must ensure that the auditing of that action performed must be atomic.”
What do you understand that to mean?
John Simpkins: So “atomic” is a database terminology, so you write all the transactions or they all roll back. You don’t have partial transactions written.
Mr Beer: “There also needs a level of obfuscation to ensure that the audit mechanism is robust.”
What do you understand that sentence to mean?
John Simpkins: No idea.
Mr Beer: “The above-mentioned requirements suggest that there is a need for a correction tool to be delivered which performs the correction, audits it and saves both changes.
“A simple low-cost solution for the tool is to provide a Linux shell based utility …”
Can you help us with what Linux was please?
John Simpkins: It’s an operating system that they have used on – well, on the branch database.
Mr Beer: “… which calls a PL/SQL package …”
Can you explain what that is, please?
John Simpkins: A programme language SQL is a way of writing structured query language transactions to an SQL database, which is the branch database is.
Mr Beer: “The package will allow inserts to the following transactional tables in the Branch Database Live schema with the exception of the Message Journal. All inserts will be audited in the table”, and then a reference is given.
Then the question that Mr Hodgkinson asked:
“From the above we wish to clarify, with evidence where possible:
“How does this process operate and who has the ability to be able to perform this, eg POL and/or Fujitsu?”
Then secondly:
“What monitoring is performed over the table”, and then the reference is given.
If we can go back, please, to page 4, we can see Dave King’s response. He was the senior technical security assurance manager. What part of the Post Office was that within, to your knowledge?
John Simpkins: I don’t know.
Mr Beer: So this is still within the Post Office at the moment, and he says:
“… I believe the only way we will be able to resolve this is if you get confirmation from Fujitsu of whether this has ever been done and what the process is (POL have no direct access to the database).”
Does that sentence in the brackets there correspond with your understanding, that POL had no direct access to the database?
John Simpkins: Yes.
Mr Beer: “If corrections are needed, ‘we’ insert a transaction to correct the situation following a reconciliation process rather than make direct changes to any transaction in the database.”
Then raises an issue about a contact within Fujitsu.
Can we go back to page 1, please, of the email chain and then if we go to the foot of the page – keep going, keep going. Thank you.
At the very foot of the page we can see an email from you to James Davidson of 15 May 2014. Who was James Davidson?
John Simpkins: I don’t know. I was asked by someone to provide some technical input from a couple of questions, so I did.
Mr Beer: You say:
“… we did not discuss timescales but I have just been asked by Leighton for some more details before a 10.30 meeting today.”
Who was Leighton?
John Simpkins: I can’t remember, I’m afraid.
Mr Beer: At this stage, you’re saying “I have just been asked by Leighton for some more details before a 10.30 meeting”, and it is 10.24 when you are writing the email. Did you have sufficient time to prepare the answers or are you hinting that you hadn’t?
John Simpkins: I probably was hinting that I have been given a very tight deadline, so I have not researched this information as thoroughly as I probably could.
Mr Beer: Did you know what the answers that you were giving were going to be used for, ie the purpose to which they were going to be put?
John Simpkins: No. I was very surprised to read the Deloitte –
Mr Beer: I’m sorry?
John Simpkins: I was very surprised to read the Deloitte – the references in there to this email.
Mr Beer: Why were you surprised of what became of the answers?
John Simpkins: Because I was just asked a couple of technical questions. I mean, I don’t mind the answers being there, but no one told me where they were going to go.
Mr Beer: What, if anything, would you have done differently if you knew where the answers were going to go and what use was going to be made of them in the future?
John Simpkins: I would have missed the 10.30 deadline.
Mr Beer: What other research would you have undertaken?
John Simpkins: I would have talked to the database – the database architect.
Mr Beer: Who was that?
John Simpkins: Gareth Seemungal.
Mr Beer: Say that again please?
John Simpkins: Gareth Seemungal.
Mr Beer: So if we look then, question 1, about the – and then there’s a reference to the table – and then you have broken down the question, part 1:
“How does this process operate and who has the ability to be able to perform this, eg POL and/or Fujitsu?”
What did you understand the question to mean?
John Simpkins: It’s talking about the branch transaction correction utility, and so I was trying to – I know it has been used once, so I was using that information to try and detail what was the process, how that time had come about.
Mr Beer: You answer it as follows:
“The normal support route is used to identify when a fix is required either from a branch raised incident or estate monitors that alert support staff.
“An TfS incident would be raised with evidence.”
What does a “TfS incident” mean?
John Simpkins: So TRIOLE for Services is the first line helpdesk used at this time.
Mr Beer: Who would raise that incident?
John Simpkins: So that would be – it depends on where the issue was identified. It could have come from the branch – MSU, it could have come from a postmaster or from SMC, or from – in Post Office.
Mr Beer: You say:
“This would be transferred to the SSC as a PEAK because they support the applications.”
Who is the “they” in that sentence?
John Simpkins: SSC.
Mr Beer: “The SSC would investigate with evidence from the support branch database and then liaise 4th line development (evidence and progress would be recorded on the PEAK).
“4th line development would generate the required scripts using a test system to make the correction. An MSC …”
What’s an MSC?
John Simpkins: Managed service change, so it’s part of the authorised changes to systems. We used to have OCPs and then it became MSCs and now it’s TfS, and they’re all changes.
Mr Beer: Overall, what is that describing, an MSC or –
John Simpkins: It’s going to describe what the change is and it’s going to go to people to be authorised. It’s going to – this goes to a distributed list who have to authorise it.
Mr Beer: So:
“An MSC … would be raised for permission to run the support tool on the live branch database.
“The SSC would run the script using the support tool against the live estate.”
So, overall, in this part of the answer, you’re describing who has the ability to perform the function and it is generated by either subpostmasters, through first line support, or somebody within Fujitsu themselves. It’s picked up by third line support and, if it’s necessary to run scripts using a test system, a request would be raised for permission to do so?
John Simpkins: Yes.
Mr Beer: Is that a fair summary?
John Simpkins: That’s a fair summary.
Mr Beer: The second part of the question that you have broken down:
“What monitoring is performed over the table …”
Can you explain, first of all, what the question means, “What monitoring is performed over [that] table”?
John Simpkins: That table is the journal that this tool writes to, so I’m presuming it was meaning how is that table populated and then does it go anywhere else, audit or whatever.
Mr Beer: You answer:
“The Support tool is written to run under the SSC (read only role) …”
What does that mean?
John Simpkins: So the roles – it doesn’t have permission to write to database.
Mr Beer: “… and connects internally as the APPSUP role (write permission).”
What does that part of the sentence mean?
John Simpkins: That’s the database role that does have permission to write to the database.
Mr Beer: What does “and connects internally” mean?
John Simpkins: It means that we don’t manually have to switch the role to APPSUP. The tool does it all internally. If we needed to switch role to APPSUP we have to request that permission from the SecOps team and the SecOps team get the ops team to make the change and then we can switch role to APPSUP.
Mr Beer: What was “APPSUP”?
John Simpkins: APPSUP is the role that allows write to – update to the database.
Mr Beer: What does “APPSUP” refer to?
John Simpkins: Application support, I presume, but that’s …
Mr Beer: Why would operational security ordinarily be required to be contacted to give permission to use APPSUP?
John Simpkins: So this was a security – an additional check to make sure that the reason we’re requesting write update to the database is reasonable.
Mr Beer: But this allowed an automated access to the APPSUP role?
John Simpkins: Correct, so normally APPSUP would be – we would use APPSUP when there is no tool – tooling defined for it – for when there is no plan. This is a planned tool. This tool can do all the connections underlying.
Mr Beer: You say:
“All changes are written to the AUDIT logs.”
What do you mean by that answer?
John Simpkins: I believe that the output from the tooling is written to a log and then that log is written to the audit database.
Mr Beer: You say:
“The output from the support tool is captured and recorded on the PEAK.”
John Simpkins: That’s – yes, we did that as well, but there’s – that is a manual process.
Mr Beer: But you’re saying that there’s a footprint of the use of the tool written automatically to the audit log?
John Simpkins: Correct.
Mr Beer: “I can find just one recorded use of this tool”, and then you set it out.
John Simpkins: Yes.
Mr Beer: Then over the page, please, you say:
“This indicates that this parameter has not been changed since created on [5 October 2009].”
John Simpkins: I think that was going from there is no update time stamp but there is a creation time stamp, that’s what I was going from there.
Mr Beer: What do you mean by “This indicates that this parameter has not been changed”; what are you referring to, the parameter?
John Simpkins: It would be a specific question about a database parameter and that is the output of my query against that parameter: what are the fields on that database parameter?
Mr Beer: What are you saying by that sentence?
John Simpkins: So I’m detailing the settings of that parameter and making an observation that I believe it hasn’t been updated since creation.
Mr Beer: You’re saying it has only been used once?
John Simpkins: No, sorry, that is a separate query to the other. There was two queries. One was about the actual tooling and has it been used and then there’s another query about this parameter.
Mr Beer: Yes, if we just go back to the foot of the previous page and up a little bit, it’s the bullet point in bold:
“Can we see evidence to demonstrate that this parameter is currently set to ‘True’?”
What does that question mean?
John Simpkins: I’m unaware. I was looking at what the parameter is in that data – so that question is – sorry, “There is a parameter in the database, it’s in this table, can you find out is the value true?”
Mr Beer: What does that mean though?
John Simpkins: I – how that parameter is used, I cannot tell you.
Mr Beer: You just wanted – you answered the –
John Simpkins: I answered the question, the absolute question: “What is that parameter set to?”
Mr Beer: Overall do the answers mean that the only way that someone in the SSC could amend cash accounts was by using the process that you described or were you saying that that’s just one type of process for amending cash accounts?
John Simpkins: Overall, I was answering the question about the usage of that tool, which was the question. I would say there is the ability of direct access, but that is extremely difficult. That is the reason why there is a tool for doing such, and why – there’s many tables that are written to in the branch database, not just a central database table with the branch details – the cash account details, or the BTS details in this time, and you have to update all the correct tables in the right order or atomically, and this is a tool that is designed for that and the – actually, the fourth line team would devise the scripts to be executed to do it correctly.
Mr Beer: Would it be wrong to say that, overall, from this email, you were saying that cash accounts have been amended only once?
John Simpkins: I think it is a fair statement because I think of how difficult to update a cash account – a branch trading statement in HNG-X database is.
Mr Beer: So that would be a fair statement: you were saying that cash accounts, to your knowledge, had only been amended the once and that was referring to the entirety of the period of time that you had worked in the SSC?
John Simpkins: We’re talking about the branch database, we’re talking about HNG-X from 2010 to now.
Mr Beer: Yes.
John Simpkins: Yes.
Mr Beer: Sir, that would be an appropriate moment for the morning break.
Sir Wyn Williams: Very well.
Can I just ask, Mr Simpkins, so that I’m clear about this, so in the last series of questions and answers, from Mr Beer and your answers, you are confining what you say to the time from the rollout of Horizon Online, as opposed to Legacy Horizon? You’re not saying anything about Legacy Horizon?
John Simpkins: Correct, this is talking about the branch database, which is only used from HNG-X.
Sir Wyn Williams: Fine, I’ve got it. Thank you, yes.
Right, quarter of an hour, Mr Beer?
Mr Beer: Yes, so 11.30, please, sir.
Sir Wyn Williams: Fine.
Mr Beer: Thank you.
(11.15 am)
(Short Break)
(11.30 am)
Mr Beer: Good morning, sir, can you see and hear me?
Sir Wyn Williams: Yes, I can.
Mr Beer: Thank you. Mr Simpkins, just one question arising from the last answer you gave. You said to the Chairman that your email should be read in the context of only referring to Horizon Online.
John Simpkins: Yes.
Mr Beer: You said “in relation to the branch database”. What did you mean by reference to the “branch database”?
John Simpkins: The branch database is only used in Horizon Online. It wasn’t in existence, it didn’t exist in Horizon Legacy.
Mr Beer: That was something maintained by Fujitsu, it wasn’t in the branch?
John Simpkins: That’s correct, so, yes, the branch database is in the data centre.
Mr Beer: Thank you. Can we look, please, at POL00029750. You will see that this is a draft Deloitte report of 23 May 2014. If we can skip to page 3, please, and then just look at the first couple of paragraphs:
“As outlined to us by the Post Office Limited … litigation team, ‘POL is responding to allegations from subpostmasters that the ‘Horizon’ IT system used to record transactions in POL branches is defective and that the processes associated with it are inadequate (eg that it may be the source and/or cause of branch losses). POL is committed to ensuring and demonstrating that the current Horizon system is robust and operates with integrity, within an appropriate control framework’.
“POL is confident that Horizon and its associated control activities deliver a robust processing environment through three mechanisms: POL have designed features directly into Horizon to exert control; POL operates IT management over Horizon; and POL have implemented controls into and around the business processes making use of Horizon. Collectively these three approaches of inherent systems design, ongoing systems management and business process control are designed to deliver a Horizon processing environment which operates with integrity.”
Then further down the page, please:
“Deloitte has been appointed to:
“consider whether this Assurance Work appropriately covers key risks relating to the integrity of the processing environment,
“to extract from the Assurance Work an initial schedule of Horizon Features,
“to raise suggestions for potential improvements in the assurance provision.”
Then it sets out how it is going to do its work. Were you aware that this process was being undertaken in 2014?
John Simpkins: No.
Mr Beer: Can we look forwards, please, to page 38. I have just shown you those initial parts of the document in order that you can understand what the document is and the bit that we’re going to look at where it falls within it. As part of their assurance work, Deloitte produce an assurance schedule and they say that they:
“… present below a schedule of the Assurance Work and sources we have identified which relate to certain groups of Horizon Features.”
They record an assessment of the level of comfort that POL has over the relevant Horizon feature. Do you see?
John Simpkins: Yes.
Mr Beer: Then if we can scroll forwards to page 48, please. Can you see under the “Area”, “Usage”, in the second box down “Branch Ledger transactions are recorded accurately in the Audit Store”, as the assertion giving rise to process integrity?
The description of the feature of processing integrity is said to be:
“Formalised change control approval and monitoring process over usage of Balancing Transactions”.
The source of that is said to be an email communication from you of 15 May 2014. That’s the thing we looked at and “articulating control design around this process”, and the “Level of Comfort” that POL are said to have had is “Partial”.
Then the next row, the “Key Assertion” giving rise to process integrity was:
“Branch Ledger transactions are recorded accurately in the Audit Store.
“Description”:
“Audit trail monitoring the usage of balance transactions.”
Again, the same source of evidence. Did you know that your email was going to be used in this way?
John Simpkins: No.
Mr Beer: What, if anything, would you have done differently in terms of its construction and the contents of it if you had known it was going to be used in this way?
John Simpkins: I think I said earlier I would probably have had a talk to the database architect just to clarify that this is – my email answered these questions. But I was fairly happy with what I replied to for the two questions that I was asked.
Mr Beer: So am I detecting this, that it was the narrowness of the answers that you gave –
John Simpkins: Yes –
Mr Beer: – that if you had known they were going to be used for this purpose you might have added more to them?
John Simpkins: Yes.
Mr Beer: I take it, therefore, that you didn’t discuss with Deloitte the provision of your email or the content of the answer?
John Simpkins: Definitely not.
Mr Beer: Can we look, please – that can be taken down – at POL00028070. We are three years on now and another report, also in draft, from Deloitte. If we go again to page 3, please, you will see a summary from Deloitte of the Horizon Online system. It sets out the controls that respond to the fundamental risks under those subparagraphs.
Can you recall this report being produced?
John Simpkins: No. I have seen it in my bundle, but I don’t recall it being produced.
Mr Beer: Do you recall whether they, that’s Deloitte, spoke to you about it, the contents of the report?
John Simpkins: No.
Mr Beer: Can we just look forwards, please, to page 83 of the document, please. In an appendix, they set out a list of individuals that they, Deloitte, say were interviewed and can you see your name two from the bottom here –
John Simpkins: I can.
Mr Beer: – “John Simpkins, SSC team leader”. Were you interviewed by Deloitte?
John Simpkins: I don’t recall being interviewed by Deloitte, no.
Mr Beer: You would probably remember if you were, wouldn’t you?
John Simpkins: I would have thought so.
Mr Beer: So this is incorrect?
John Simpkins: They have also got Jon Hulme as working for Post Office.
Mr Beer: I’m so sorry?
John Simpkins: Sorry, the one above is incorrect as well.
Mr Beer: Ie his employer ought not to be POL?
John Simpkins: Is Fujitsu, yes.
Mr Beer: So, in any event, as far as the content of the October 2017 Bramble report for Deloitte, you were not interviewed for that?
John Simpkins: I don’t recall ever being interviewed for that.
Mr Beer: That can be taken down, thank you.
Can we look, please, at FUJ00088036. If that can just be expanded a little bit, please.
Do you recognise this?
John Simpkins: Yes.
Mr Beer: What do you recognise it as?
John Simpkins: It’s a support – well, it’s a design document for when we were introducing OpenSSH to remotely access the counters.
Mr Beer: So we’re here dealing with Legacy Horizon, as it became known –
John Simpkins: Correct.
Mr Beer: – not Horizon Online? You would have been, I think, provided with this at the time, or seen it at the time, or had access to it at the time?
John Simpkins: We would have had access to it. We – the SSC were generally on a standard distribution list to comment on documents and give feedback to documents but they were routed out amongst the team. I don’t know if the dimensions, or if – this was probably PBCS(?), I don’t know if that contains the reviewer’s comments to see who –
Mr Beer: If we skip forwards, and then go down, is that what you are referring to, the reviewer’s details, ie those that were given the opportunity to review?
John Simpkins: That’s correct, yes. So you’ve got mandatory – you’ve got Mik Peach and he was just the figurehead for the document reviews. They would be sent to the SSC and then given to someone.
Mr Beer: Then Mr Peach underneath him, I think?
John Simpkins: Yes.
Mr Beer: Sorry, Mr Parker underneath him?
John Simpkins: Yes.
Mr Beer: Thank you. So this would have been a document that the SSC had an opportunity to review and comment on and then, in its final iteration, distribute it to the members of the SSC?
John Simpkins: No, it would be put in dimensions storage. We may put it onto our SSC website some – if it were the – if the final version were sent to us, this is the type of document we would put on the SSC website, so it’s searchable.
Mr Beer: So members of the SSC would have access to it?
John Simpkins: Correct.
Mr Beer: Thank you. Can we just go to page 9, please, and look at the introduction to see what the document is. Under 1.1.1, “General”:
“[SFS] …”
I think that’s “security function specification”; would that be right?
John Simpkins: I don’t know.
Mr Beer: If I’m right that that is what SFS means, security functions specification, what was the security function specification?
John Simpkins: I don’t know.
Mr Beer: Anyway it, assuming that it is what I say it is:
“… mandates the use of Tivoli Remote Console … for the remote administration of Data Centre platforms.”
Can you explain what that sentence is saying, please?
John Simpkins: So Tivoli was a management package that was used for eventing, amongst other things, and had the ability to run some commands, and part of it was a remote console which allows you to commit to a computer in a console – a command line facility, so you can execute commands on that computer.
Mr Beer: Thank you. It continues:
“This records an auditable trail of log-ins to all boxes accessed by the user.”
Is that accurate, to your knowledge?
John Simpkins: I believe so. I didn’t manage Tivoli.
Mr Beer: It says:
“It is a matter of considerable discussion and correspondence that the [Tivoli Remote Console] is slow and difficult to administer.”
Do you remember that, ie that it was slow and difficult to administer?
John Simpkins: Not particularly.
Mr Beer: “This has led over time to BOC personnel …”
BOC, can you help us with what that was?
John Simpkins: No.
Mr Beer: Maybe Belfast Operation Centre?
John Simpkins: Could be.
Mr Beer: If it is Belfast Operation Centre, what was the Belfast Operation Centre?
John Simpkins: They were the operations people, so –
Mr Beer: So part of Fujitsu in Belfast?
John Simpkins: Correct, yes, they looked after the data centres.
Mr Beer: “… relying heavily on the use of unauthorised tools (predominantly Rclient) …”
What was “Rclient”?
John Simpkins: That was a remote client so that’s another tool that you can use to get a command line interface onto a server remotely. So that’s what – I remember we did use that to connect to the counters.
Mr Beer: You used that as well, did you?
John Simpkins: We used that to connect to the counters.
Mr Beer: To connect to counters?
John Simpkins: Correct.
Mr Beer: “… to remotely administer the live estate. Its use is fundamental for the checking of errors.”
Would you agree with that sentence?
John Simpkins: Yes.
Mr Beer: “The tool does not however record individual user access to systems but simply records events on the remote box that Administrator access has been used.”
Does that reflect your understanding?
John Simpkins: Yes, so – yes, you would probably have a Windows event that that user has been granted authorisation to connect to the box, so a security event, I would imagine.
Mr Beer: But it doesn’t record what happened?
John Simpkins: It wouldn’t record – yes. It wouldn’t record –
Mr Beer: It was fact of access but not –
John Simpkins: Or even who did it. It would have been under a generic user.
Mr Beer: So it doesn’t record what the purpose of the access was or what was done in the course of access and it doesn’t record who has access. As you say, it would be a generic record?
John Simpkins: Yes.
Mr Beer: “No other information is provided including success/fail so it is not possible to simply audit failures. The use of such techniques puts Pathway in contravention of contractual undertakings to the Post Office.”
Do you remember that issue arising back when using Legacy Horizon?
John Simpkins: Not particularly. I do remember we used Rclient. I don’t particularly remember the Tivoli remote console, but I don’t remember particularly using it, and then –
Mr Beer: Do you remember an issue being raised as to the SSC’s use of Rclient putting it in breach of its contractual obligations or undertakings to the Post Office?
John Simpkins: I don’t particularly remember that but I do know that we did switch to using OpenSSH to connect.
Mr Beer: “After proposals in this SOD …”
I’m afraid I couldn’t find what that meant: “SOD”?
John Simpkins: The system support – outline design, that’s what – this document, is it?
Mr Beer: Ie this very document?
John Simpkins: Yes.
Mr Beer: The system outline design?
John Simpkins: Yes.
Mr Beer: I’ve got it. So:
“After the proposals in this [document] have been implemented a CP …”
Can you help us with that?
John Simpkins: Change proposal.
Mr Beer: “… will be raised to phase out [Tivoli Remote Console] …
“This document provides an outline design, which primarily stops Pathway being in contravention of its contractual undertakings but also provides an acceptable and agreed level of secure access to systems for support activities.”
Can you help us with what, if any, relationship the BOC – if I’m right, the Belfast Operation Centre – had to the SSC?
John Simpkins: So they looked after the data centre systems, so the operating system of the data centre servers, the databases in the data centre. So if it wasn’t written by Pathway, they generally looked after it; if it was written by Pathway, we looked after it, if that makes sense.
Mr Beer: I think I understand. Can we go to page 13, paragraph 4.1.2, please. Can we just scroll down a little bit. I should read 4.1 first, “Areas of concern”:
“There are two major areas of concern with the current support processes:
“Second line support does not have the tools necessary to perform their function …
“Third line and operational support organisations access to the live system is not fully audited and in some cases is restricted in the actions that can be carried out;
“The consequences of these two issues are specified in the following sections.”
Then under 4.1.2:
“Third line support staff receives repeat instances of calls that should have been filtered out by second line. Handling repeat calls is not an effective use of third line support resource.
“The current support practices were developed on a needs must basis; third line support diagnosticians had no alternative other than to adopt the approach taken given the needs to support the deployed Horizon solution.
“The consequences of limited audit and system admin access afforded to third line support staff provide the opportunity to:
“Commit fraudulent acts;
“Maliciously or inadvertently affect the stability of the new Network banking and Debit Card online services;
“In addition a complete audit would allow Pathway to defend the SSC against accusations of fraud or misuse.”
Again, in 2002, did you know that this was an issue?
John Simpkins: I was unaware that this was an issue.
Mr Beer: Did you know that an investigation or a review was being undertaken into the extent of third line support access and the method that the SSC was using to procure such access and that it was said to have provided the opportunities set out there?
John Simpkins: Not particularly. I do remember we were talk – I remember us talking about the OpenSSH access and I also remember it being told that it was going to record every key press. So I knew that there was enhanced audit in what we were moving to but I don’t remember particularly that it was put to us in this way. It was – yes, it was enhanced audit. I did know that was coming in.
Mr Beer: Can you repeat that last sentence, I didn’t hear it?
John Simpkins: It was enhanced auditing and, in this new method of access, I knew that was coming in.
Mr Beer: So you knew that a new method of access that was more auditable –
John Simpkins: Correct.
Mr Beer: – was being introduced, you didn’t know the reasons that sat behind it?
John Simpkins: Yes, so, obviously, I can infer something has come in that’s more auditable, the old one obviously was not auditable enough.
Mr Beer: Would you agree with what is said here as to the reasons for its introduction, namely that the type of access that was afforded did give those opportunities?
John Simpkins: I don’t know if I agree with the first one.
Mr Beer: That it didn’t give the facility to staff to commit fraudulent acts?
John Simpkins: Yes, I’m – as far as I’m aware, the APS transactions and banking transactions were all digitally signed. So I can’t see how SSC would be able to do any fraudulent activities there.
Mr Beer: The second one, maliciously or inadvertently –
John Simpkins: I imagine maliciously, you could try and damage a database or take down an agent which would cause an outage, or VPN server. So yes, I could see maliciously.
Mr Beer: We can put that to one side. Can we look, please, at FUJ –
I’m so sorry, we should have looked at one other passage in that document. 4.3.2 on page 15, please. Thank you. The authors record that:
“All support access to the Horizon systems is from physically secure areas. Individuals involved in the support process undergo more frequent security vetting checks.”
Were those two things accurate?
John Simpkins: Yes.
Mr Beer: The site was physically secure and there was some enhanced vetting?
John Simpkins: Yes, so we had security checks on all the staff. The site – the room on the sixth floor had its own pass system. It wasn’t part of the general building pass system. The – we had separate computers for connecting to the data centre, as well as your corporate system. It was on a totally separate system. You had separate passwords. You had two factor authentication with secure IDs. So, yes, it was fairly secure.
Mr Beer: Then it says:
“Other than the above controls are vested in manual procedures …”
That doesn’t make complete sense:
“… requiring managerial sign off controlling access to post office counters where update of data is required.”
It’s difficult to understand exactly what that means.
John Simpkins: It’s probably talking about the OCPs and OCRs and the MSCs, and things we were talking about, where there were other sign offs, but that was a manual sign off to give you authorisation, but it didn’t physically stop you doing it without that.
Mr Beer: And there was no audit of it?
John Simpkins: Correct.
Mr Beer: “Otherwise third line support has:
“Unrestricted and unaudited privileged access … to all systems including post office counter PCs …”
That was true, yes?
John Simpkins: Yes.
Mr Beer: “The ability to distribute diagnostic information outside of the secure environment; this information can include personal data (as defined by the Data Protection Act), business sensitive data and cryptographic key information.”
That was true as well?
John Simpkins: No.
Mr Beer: No? In which respects was it false?
John Simpkins: So we didn’t support the KMA – we didn’t support the key management. We supported its interactions, but we didn’t support it – that was where the key material was, I believe, and we didn’t support the audit server either, so we didn’t have access to those. We had – there was a separate key server, which was in a little room that was locked and used by the security people. There was a KMA work station, which was used by a fourth line support person who did the support for the key management. So there were areas we didn’t support.
Mr Beer: Right, so it’s an accurate statement but needs to be qualified, in that there are some areas that it does not apply to?
John Simpkins: Yes.
Mr Beer: Is that a fair way of describing it?
John Simpkins: Specifically, I’m thinking about the cryptographic key information.
Mr Beer: Skipping a paragraph, which is a repetition largely of what appeared previously, the authors record:
“There are … no automatic controls in place to audit and restrict user access. This exposes Fujitsu … to the following potential risks:
“Opportunity for financial fraud …”
Would you agree with that?
John Simpkins: No, I don’t see how you could do financial fraud.
Mr Beer: “Operational risk – errors as a result of manual actions causing loss of service to outlets …”
John Simpkins: Yes.
Mr Beer: You agree with that?
John Simpkins: Yes.
Mr Beer: And:
“Infringements of the Data Protection Act.”
John Simpkins: Yes.
Mr Beer: You would agree with that, thank you.
Now, this process that’s being described, ie the backward look and the fixes that were proposed, you didn’t include any of that in your email of May 2014?
John Simpkins: No.
Mr Beer: Is that because you were answering the narrow question that was asked of you?
John Simpkins: There were literally two questions and I answered them both.
Mr Beer: Can we look at FUJ00089756.
John Simpkins: This also is –
Mr Beer: This is Legacy?
John Simpkins: – Legacy and the questions were in –
Mr Beer: They don’t say Horizon Online but they could only apply to Horizon Online?
John Simpkins: Exactly.
Mr Beer: Can we look, please, at – yes, thank you, we’ve got it up.
This is a PEAK, PEAK number 0208119. You will see if we just scroll down a little bit, please, and a bit more, that it’s opened in February 2011?
John Simpkins: Yes.
Mr Beer: I think you were aware of this PEAK because it related to your work and, at one stage, I think it was referred to you and you made a contribution to it. I think we can just see that if we go forward to page 3 and just scroll down. I think we can see an entry on there of 17 August 2011 by you. Yes?
John Simpkins: Yes, so this is about the APPSUP.
Mr Beer: Yes, so if we just go back to the beginning then, please, page 1, and the summary of the incident we can see is that:
“SSC Database users do not have correct permissions.”
Can you see whether this was raised by somebody within Fujitsu or –
John Simpkins: Yes, it is “Call Logger”, top right, by Mark Wright in the EDSC.
Mr Beer: Then if we scroll down to the impact statement:
“SSC users affected have more access than is required to database resources. This is contrary to security policy.
“… There is currently no ‘cost’ to this issue.”
As for “Perceived Impact”:
“… The customer is not aware of this problem or change.
“Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and not expected).”
Then if we can go down, please, to what Mr Wright wrote when opening the PEAK “Summary”, which we have seen above:
“Database users do not have correct permissions.”
Then in more detail – and we’re dealing with Horizon Online here, aren’t we?
John Simpkins: Yes, we are.
Mr Beer: “Development have delivered scripts to allow SSC users to perform certain tidyup tasks (like clear failed recoveries). However they have been delivered to work against an SSC role which SSC users have not been granted as SSC users have the APPSUP role.”
Can you explain what that first paragraph means, please?
John Simpkins: So these are roles in the database that grant different permissions. So the SSC role is a read only role, so that’s our default role. The APPSUP role is the one we were talking about before which does have the update permissions.
Mr Beer: “Either SSC user creation/configuration needs to be amended to make sure we have ALL required permissions of …”
Then I think that’s meant to be “or”:
“… [or] the scripts will need amending to match how our users are set up in live.”
John Simpkins: Yes.
Mr Beer: Again, can you decode that for us, please?
John Simpkins: So the scripts are obviously using a different permission that does no longer work and either the SSC profile user on the database has to be updated or the scripts have to be updated, so they work.
Mr Beer: Then if we scroll down, please, he, that’s Mr Wright, I think, includes an email chain that’s included. If we scroll down a little further – thank you – I think we can see an email from Anne Chambers of 1 February 2011 that’s been cut into this PEAK. Can you see that?
John Simpkins: Yes.
Mr Beer: She says:
“Unfortunately development write their scripts explicitly to use SSC. So I think we’re stuck with it unless they deliver new scripts (which would not be a popular or quick option).
“When we go off piste we use appsup. Can we have both??”
Firstly, can you help explain what the first paragraph of Ms Chambers’ email is referring to?
John Simpkins: So I think that’s talking about the scripts that Mark was detailing above, like the failed recovery tidy script, that there you write them to use the SSC profile, which now no longer has write permission.
Mr Beer: Then she says:
“When we go off piste we use appsup.”
What does that mean?
John Simpkins: So, like we were just talking about the script, that script is written to – it’s a known issue about clearing a failed recovery once they have been investigated. “Off piste”, she is basically saying that there is no tool to do this, this is something we have not come across before, therefore you could wait and write a tool to do the correction, or we have to go in manually to do the correction.
Mr Beer: And we use APPSUP to do that?
John Simpkins: APPSUP is the write role, the role with the update permissions.
Mr Beer: What do you understand the reference to going “off piste” to mean?
John Simpkins: Where there is a new issue that you haven’t got a script to fix already.
Mr Beer: Mr Gibson replies:
“I suspect you can have both but either way you need a development fix as they produce the user creation script which does the database bit. If they have to produce a fix, I’d advise making one of the roles suitable rather than having a mix of grants across both roles.”
Then scroll up, please. Mr Wright replies:
“I thought the original issue was why have the SSC users not had the SSC role granted? If it is a bug in the creation scripts then yes, needs [development] to fix but I thought something was said the other day about the SSC users not being set up correctly at the start?”
What is he referring to there?
John Simpkins: So I think this is about the SSC users not having the permissions to switch to the database roles, so that they couldn’t run – the script should automatically switch to whatever role it needs to do in the script and it wasn’t. Then he is saying “Are the SSC users set up correctly? Are the permissions correct for the SSC user?”
Mr Beer: Then if we go forward a page to your contribution. Scroll down, please. Six months on, you say:
“This is getting confused, this incident is about the SSC role which ISD …”
“ISD” being?
John Simpkins: They are the operations people.
Mr Beer: “… need to give to the SSC in order to run a script provided to the SSC by development.”
Then underneath that it seems you transferred the call to a different team; is that right?
John Simpkins: Yes, there’s the host – “APOP-Host-Dev”, so APOP is a database development team.
Mr Beer: Why was it necessary to transfer?
John Simpkins: I think it was because I really needed an answer about the database roles and what they should be set as.
Mr Beer: I’m not going to carry on through the PEAK, save to go to the last page, please. We see Mr Haywood. We’re sort of a year and three months on from the start; who was Mr Haywood?
John Simpkins: The security manager.
Mr Beer: “The Business Impact has been updated:
“SSC users affected have more access than is required to database resources. This is contrary to security policy.”
Then we see him including there the impact statement that we read originally. Can you remember what the solution was to this?
John Simpkins: This is, I mentioned before, where we don’t have any default access to write permissions. I think this is the outcome from this, so we have to ask SecOps to ask ISD, the operations people to grant that permission for a temporary process, while we do the off piste things. So I think that was the output of this.
Mr Beer: When was that solution put into your memory?
John Simpkins: After this.
Mr Beer: So some time after June 2015?
John Simpkins: Yes.
Mr Beer: So does it follow that, between the rollout of Horizon Online in, say, 2010 until mid-2015, there was off-piste access by the SSC?
John Simpkins: There was. It still wasn’t the default role because the default role is read only, but you could – without going through SecOps and ISD – do set role APPSUP to be granted the update permission.
Mr Beer: How frequently was that done?
John Simpkins: Not very frequently, to my knowledge, but again you could go through the PinICLs and PEAKs to find out at that time. Sorry, OCPs and OCRs, as well, would have been …
Mr Beer: Was it, other than by looking at PEAKs where somebody had recorded that they had done this, auditable?
John Simpkins: I believe so. I believe there was –
Mr Beer: How was it auditable?
John Simpkins: Again, I didn’t support audit but I believe that it wrote a message saying that you had switched role.
Mr Beer: So you believe that you personally wrote a message?
John Simpkins: No, no, sorry, the system.
Mr Beer: The system wrote a message?
John Simpkins: The system writes a message to Audit saying that this user has switched role to APPSUP. I believe, again, that I think I saw a list of that in the GLO.
Mr Beer: Was that via a Tweet or –
John Simpkins: No, no.
Mr Beer: – or actually seeing the evidence?
John Simpkins: I think I saw the evidence of a list of the times that they switched into it.
Mr Beer: Was it known within the SSC community that this going off piste using APPSUP was problematic?
John Simpkins: We didn’t know it was against any rules that Mr Haywood knew but going off piste, as it was put, would definitely require an OCR or OCP to be raised and signed off by SSC manager for OCRs and others for OCPs.
Mr Beer: That requires the person that’s going off piste to tell somebody else that they’re doing it?
John Simpkins: Yes.
Mr Beer: It puts the onus on the individual?
John Simpkins: Yes. There were procedures in place and Mik was very sure about his procedures and we had two sets of eyes procedures as well for doing such things.
Mr Beer: If that was the case, that there were procedures in place that included two sets of eyes on it, do you know why a change was necessary?
John Simpkins: I would say to make doubly sure that we couldn’t do it. It’s another step – there is an idea of six steps of separation, where you could – like another team can’t do certain things, we can’t access audit, we can’t access the KMA, and that’s a security put in and this is another one of those.
Mr Beer: Again, in your May 2014 email, why would you not tell those that were asking about this –
John Simpkins: I was literally asked two questions and I literally replied to those two questions.
Mr Beer: So if you had been asked the question “Look, we’re looking at the extent to which the SSC can do things to data without there being a proper security control mechanism in place or an automatically generated audit trail of them, can you tell us about any of those things, please?” you may have mentioned what we’re talking about now?
John Simpkins: And I would probably refer them to the audit architect because we don’t support audit, so I couldn’t really tell you that much about what does get written to audit, where it gets written.
Mr Beer: No, but what you could say is that “We have spent, by then, four years going off piste” –
John Simpkins: I could say that for four years we have had the access to switch role to APPSUP and these are the – probably the times we have done it, based on the PEAKs and OCPs/OCRs.
Mr Beer: Of course, when you were making your contribution to this chain, that was in August 2011 –
John Simpkins: Yes.
Mr Beer: – to this PEAK?
John Simpkins: Yes.
Mr Beer: Did you then drop out of the PEAK thereafter?
John Simpkins: I think I rooted it off to a different team at that stage.
Mr Beer: So you weren’t aware of, necessarily, what happened in the administration of the PEAK thereafter?
John Simpkins: Not particularly. I would have known that there was a procedural change when it was changed and this is the new process we got to follow to get access to APPSUP.
Mr Beer: But, back to the May 2014 email, it was the narrowness of the questions that you were asked that caused the narrowness of the answer?
John Simpkins: I was only asked two questions so it was exactly that.
Mr Beer: Can we turn, lastly, to some EPOSS faults, please. Can we look, please, at FUJ00036863. I think you raised this PinICL?
John Simpkins: Yes.
Mr Beer: Is that right?
John Simpkins: That’s correct.
Mr Beer: Would that have originated from a subpostmaster call?
John Simpkins: No.
Mr Beer: Where would it have originated from? Where did it originate from?
John Simpkins: It originated inside the SSC.
Mr Beer: And how?
John Simpkins: I don’t know how I found that there were null modes in APS and EPS transactions – sorry, EPOSS transactions but that is the key to –
Mr Beer: How did you know to connect the problem with EPOSS?
John Simpkins: So they – we’re talking about different transaction types. APS transactions go into the APS database. They are a type of transaction, like Bill Payments, that’s a APS transaction. EPOSS transactions are a different type, like transacting the stamp or – for example, yes. So they are two different types of transactions and where they go.
Mr Beer: Can we look, please, at FUJ00058190, and can we look at page 8 of this document, please.
I think that’s a rogue reference. FUJ00058190. Yes, it’s my fault.
I will ask the questions without the document reference.
John Simpkins: Sure.
Mr Beer: The EPOSS fault that you raised, were you aware at that time that there was a serious instability issue with EPOSS?
John Simpkins: Only from what the PEAKs we were getting in, I would say. What instability in particular?
Mr Beer: Were you aware that it was proposed that there should be a rewrite of the code or at least the code as far as it related to the cash account?
John Simpkins: No, I wasn’t aware at that time.
Mr Beer: Do you remember any discussions within Fujitsu about the need to rewrite the EPOSS code as far as it related to the cash account?
John Simpkins: No, I wasn’t aware.
Mr Beer: Yes, thank you very much, Mr Simpkins. They are the only questions I ask for the moment.
I believe Mr Stein is shaking his head.
(Pause)
Mr Beer: Sir, I wonder whether we might break for a couple of minutes. Ms Page wanted to raise an issue with me and –
Sir Wyn Williams: Yes, by all means. I will stay close by, so just alert me and I will come back on screen, okay?
Mr Beer: Yes, thank you.
(12.17 pm)
(Short Break)
(12.23 pm)
Mr Beer: Sir, can you see and hear me?
Sir Wyn Williams: Yes.
Mr Beer: Thank you very much. Mr Simpkins is just being shown back into the room.
Sir Wyn Williams: Sorry, would you repeat that?
Mr Beer: Yes. Mr Simpkins is just being shown back into the room. He has taken his seat now and we’re ready to go with Ms Page first. Thank you.
Questioned by Ms Page
Ms Page: Mr Simpkins, hello. I’m Flora Page. I represent a number of the subpostmasters and, indeed, some of them were prosecuted, as you probably know, and some of them were sent to prison. So what I’m going to ask about is a few different areas of how your role might have affected them.
I’m going to start, if I may, with the third supplemental agreement. Now, that may not mean much to you. Have you heard of that?
John Simpkins: I think I may have had a supplemental agreement in here but I think it may have been the fourth, I’m not sure.
Ms Page: So it was, just to give you a little context of chronology, it was signed in January 2000, so relatively early in the national rollout. You were working then, weren’t you, in the SSC?
John Simpkins: Yes.
Ms Page: One of the issues that is clear from that third supplemental agreement is that the technical people in Fujitsu, and indeed as a result of that agreement it is clear that Post Office also knew, that there would be cash account errors caused by reference data, also caused by other technical faults and that, in some cases, they anticipated that they would only be picked up by subpostmasters phoning the call centre. Is that something that you can sort of accept from me, in terms of the interpretation of the agreement?
John Simpkins: I can accept that, yes.
Ms Page: All right. Well, were you and your team ever alerted to that?
John Simpkins: If the – we would take the calls – sorry, so they would contact either MBSC or HSH and then, if it was HSH it would, if it was a software issue, hopefully find its way to us and then we would investigate them based on that, but I don’t know about the agreement.
Ms Page: Well, obviously, you would be alerted if a subpostmaster came to you –
John Simpkins: Yes.
Ms Page: – through the lower lines of support, and you would know that you were speaking to a subpostmaster, but my question was: did anyone at Fujitsu, in your management structure or in any fashion, let you know and your team know that there would be or there could be faults, which would only become apparent because a subpostmaster alerted the helpdesk to that and that might come to you up through the chain?
John Simpkins: Not particularly. I can’t recall being told that there would be faults that only a subpostmaster may notice, but we did identify faults based off calls from subpostmasters. So it was definitely a thing we did and we did identify faults based on those calls.
If we identified a fault, we would scope the fault and, once it was recognised – and identify who was affected by that, so I think I’m saying the team knew that there were issues that subpostmasters were identifying that weren’t being picked up by automated things in the data centre.
Ms Page: All right. Well, in that case, can we please look at document number POL00028743. When it comes up, you will see that it’s a PEAK from 2001. It is sometimes quite hard to read these PEAKs. If we perhaps – can you read it? Are you able to?
John Simpkins: I can read it, yes. I think this was in my pack as well.
Ms Page: It will have been. If we look in closely at 12.58 on 14 April, it says the “pm” – I presume meaning postmaster:
“… extremely unhappy about the problems with his counters. He says he has had to pay out over £1,500 in losses that are due to these problems. He has informed POCL they can suspend him because he is refusing to make good any further losses.”
He asks for a face-to-face meeting:
“[He] feels very strongly about this and says he is willing to take POCL to a tribunal/court because of the stress he has suffered because of the problems.”
Then it says, a bit further down, in capitals:
“This call is only to be closed with the express permission of Julian Hall.”
Do you know who Julian Hall is?
John Simpkins: I don’t. This was entered from the Horizon System Helpdesk. This is their text before it gets to SSC.
Ms Page: I see. If we go on a bit further, if we go as far as page 4, please, and about halfway down we can see:
“This is an update for yesterday’s call [this is in capitals] made by the pm … PowerHelp server was down …
“Call was taken over by STSA Donna Moulds and the following information was manually logged:
“PM would like to add to the current complaint that transactions are currently appearing and disappearing on screen and also the PM’s counter printer has not been working either.
“PM had a message on screen stating [about the] transaction then the screen froze and timed out. When logged back in, the transaction was not on screen. PM rebooted the printer, and a receipt for this transaction was printed. Now the printer won’t print any receipts”, et cetera.
A bit further down, it says at 9.33:
“PM would like to add that on the 18th April … the PM spoke to Garreth from the Environmental Team. Garreth advised the PM that he will be in touch with him before the end of the month to investigate any problems. It is now past the end of the month, and still nothing has been done.”
If we carry on down a bit, please. This is at 9.35:
“PM feels the system is unreliable. PM cannot trust this system.”
He says again that he wants to speak to someone face-to-face. It is quite clear, as far as this postmaster is concerned, that he is saying that this is not his fault, he has not done anything wrong, the system is unreliable, yes?
John Simpkins: Yes, this was a phantom transactions call, wasn’t it?
Ms Page: It was, that’s quite right and, indeed, if we go down to page 10, we can see that reference to phantom transactions. I think a little higher –
Well, while we’re here we can see that it is closed down on the basis that:
“I am therefore closing this call as [it is] no fault in product.”
A bit higher up we can see, under 12 November 2001 Patrick Carroll:
“Phantom [transactions] have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon. I am therefore closing this call as no fault in product.”
But if we look further up and, in fact – I mean, you may be able to confirm it for us without us looking further up, the phantom transactions that the user is referring to were, in fact, witnessed, weren’t they –
John Simpkins: Yes, by the Romec engineer.
Ms Page: – by a Romec engineer, exactly. Yet, this later entry says “Well, we will just close this down, there’s no fault, it must be user error”.
John Simpkins: Yes, I did read through it. I don’t remember Pat Carroll researching this one. I know he did do a lot of monitoring and things like that, that’s all in the call, and I don’t know if this comment is after – for after those – those were put in place but, yes, I agree it doesn’t read well. But I can’t comment on what was the conclusion.
Ms Page: What I’m getting at here is, if you had known, if you had been told explicitly and clearly that there would be errors which could only be picked up by subpostmasters making calls and saying that they are experiencing, let’s say, phantom transactions, or whatever it may be, do you think you and your team would have been as willing to close down calls on the basis that it must be user error?
John Simpkins: I don’t know how many calls we closed down as user error without good proof. Again, that probably can be researched through the PinICLs and PEAKs. And this one was investigated extremely heavily with multiple changes made, monitoring put in, but I cannot – I agree, I cannot comment on the closure of that.
Ms Page: Well, when you say you can’t comment on it, what do you mean by that?
John Simpkins: I don’t know what investigations Pat had concluded to make that decision.
Ms Page: Was there a tendency to ascribe user error if a fault could not be got to the bottom of, as it were?
John Simpkins: I have heard that mentioned before by Mr Roll, I think, and I would hope not. I don’t think there was. Again, a retrospective review of the PEAKs and PinICLs might be able to clarify that.
Ms Page: Thank you. Could we perhaps look at another PinICL, or a PinICL rather than a PEAK. This one is FUJ00042388. This one begins on 25 February 2000. If we go down, please, to 1 March 2000, and if we look at 11.51, we see here, don’t we, that at 11.51 – Steve Warwick, he is one of your colleagues, is that right? You are there at the top.
John Simpkins: He was a fourth line support.
Ms Page: He was a?
John Simpkins: Fourth line support.
Ms Page: Fourth line support, I see. So does that suggest that you and your colleagues have then brought him in?
John Simpkins: Yes, so if you look at the fourth line it says “Please route to EPOSS DEV”.
Ms Page: Right, and so he is EPOSS DEV?
John Simpkins: Yes.
Ms Page: He says, at 11.51:
“This is identical to an issue which was raised approximately four months ago, the cause of which was never found.”
Do you know what happened when a cause was never found, as it were? Who was informed? Were you ever informed? Was your team ever given a message from fourth line support that said “There’s been no solution to this one, it’s outstanding”?
John Simpkins: I don’t know if we had that for this one, but we definitely were – we raised KELs which had a description of the problem and what we have looked at and they were used in order – in case it was raised again.
I think that there was another call later on, which he said a similar issue was caused by archiving and Riposte happening at the same time as doing the cash account.
Ms Page: If we go down to page 13, it comes back to you. Can you explain to us how it comes back to you?
John Simpkins: So Martin’s routed it back to the EDSC and Diane has passed it to me and I have passed it to the management support unit, so I think it was raised by the management support unit from the automated host detection, so the –
Ms Page: Perhaps if you can just explain, so does that mean – when you say the “automated host detection”, what’s that?
John Simpkins: So on the TPS database, it automatically checked things like cash accounts, and this was picked up – this PinICL was raised on the back of some of those alerts.
Ms Page: I see.
John Simpkins: So we are passing it back, the information on the PinICL, back to the team who raised the call at that point.
Ms Page: I see, and so when it says, a bit further down, “POCL have now agreed closure of this incident”, that’s because this is something that’s arisen on a platform and therefore the customer support people are actually liaising with POCL about it?
John Simpkins: Yes.
Ms Page: And POCL have agreed to close this down?
John Simpkins: Yes, so the MSU, the – or BSU at that time, they would – for – send corrected cash accounts to POCL, so this is what their process was, and then –
Ms Page: So this would definitely have involved amending cash accounts?
John Simpkins: It was involved in reporting the corrected cash accounts, not touching the system at all in any way, but reporting –
Ms Page: But explaining that there was –
John Simpkins: Explaining what the – why the cash accounts that would have been sent automatically were incorrect.
Ms Page: When you say “explaining” – we can look at it if you like – but from your memory, having read it, it does make it pretty clear, doesn’t it, that the problem is pretty intractable. This doesn’t appear to have resolved the problem, does it –
John Simpkins: Correct.
Ms Page: – on a root cause basis?
John Simpkins: Yes.
Ms Page: Indeed, it’s obviously involving Riposte, it’s involving the DataServer, it’s a pretty deep problem, if I can put it that way?
John Simpkins: Yes.
Ms Page: And this record does not show it having been resolved?
John Simpkins: Correct.
Ms Page: But, at the end of it, through the customer support team liaising with POCL, they have at least resolved the cash accounts, if not the problem?
John Simpkins: Correct.
Ms Page: You, presumably, have no idea what was then decided in terms of how cash accounts were going to be looked at or handled or dealt with going forward?
John Simpkins: No. I believe the management support unit would send a corrected cash account in these instances –
Ms Page: For the ones that had been found?
John Simpkins: For the – yes, and so this was picked up on the automated checks.
Ms Page: Yes, I see. It is fair to say, isn’t it, that cash account discrepancies came up a lot in what you were dealing with, didn’t they, at that time?
John Simpkins: They did.
Ms Page: Was there any forum for collating those and putting them together and saying “Here’s a lot of cash account problems, can we spot any patterns here”?
John Simpkins: One of the documents I had was – had all the issues that had been fixed, or listed them all as – and which were new ones that affected the cash accounts. I presume it was something to do with the AI – I can have a flick through, but it had a table at the back, and it seemed to indicate all the ones that – and how they were being detected.
Ms Page: But that wasn’t your document?
John Simpkins: No.
Ms Page: That wasn’t a document produced by SSC? No, all right.
Ms Page: Thank you, those are my questions.
Sir Wyn Williams: Anyone else?
Ms Patrick: Yes, sir, Ms Patrick here for Hudgells’ CPs.
Sir Wyn Williams: Yes.
Questioned by Ms Patrick
Ms Patrick: Good morning, Mr Simpkins. My name is Angela Patrick and I ask questions for another number of subpostmasters who were wrongly convicted and I’m instructed by Hudgells solicitors. You will be glad to know I only have two topics to ask you about and it’s about issues that arose in the management of bugs, errors and defects and the first document I would like us to take a look at is FUJ00081584.
You see there is a table at the top there and it looks like it is a note of a meeting and I think you can see there you can an attendee, there recorded.
John Simpkins: Yes.
Ms Patrick: Your name is a few from the bottom and right below yours is Gareth Jenkins. Can you see that?
John Simpkins: I can.
Ms Patrick: We think this is a table – you can see at the top, it is about a receipts and payments mismatch issue and the Inquiry has heard something about that and will hear something more. I think that was issue that was discovered in mid-2010; is that correct?
John Simpkins: Yes, newly into the HNG-X.
Ms Patrick: Yes, newly into the development of Horizon Online; is that fair?
John Simpkins: Yes, that’s fair.
Ms Patrick: The only reason I raise that is because there’s no date on the document.
John Simpkins: No, that’s fair.
Ms Patrick: If we can go to page 3, please, at the very top of the page and we can see there there’s an explanation about:
“The receipts and payment mismatch will result in an error code being generated which will allow Fujitsu to …”
There’s a bit more explanation but what I want to look at, at the bottom, is that paragraph:
“We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.”
So you said the bug was discovered in the period running up to the development of Horizon Online. Was this actually in the period which was running up to the acceptance of Horizon Online?
John Simpkins: I don’t know.
Ms Patrick: Okay, we can perhaps ask another witness.
Do you know why there was a delay in informing the Post Office about this bug?
John Simpkins: No, I don’t know.
Ms Patrick: Are you able to help us on where the feedback that’s mentioned there, that was going to be provided to the Post Office, could be found?
John Simpkins: Not to my knowledge, unless it was the list of the affected branches. I believe that there was a list produced and monitored for further occurrences but I couldn’t –
Ms Patrick: I think my reading of that is Fujitsu –
“We have asked Fujitsu why it has taken so long to react to this and escalate an issue which began in May. They will provide feedback in due course.”
Do you know if there was any feedback given to the Post Office about why there was such a long delay in informing them about the bug?
John Simpkins: No, I don’t.
Ms Patrick: Thank you. This receipts and payments mismatch bug, are you able to help us with your explanation, perhaps a simple explanation, of what it was?
John Simpkins: So I have read a little bit up on it, so it was when you were doing your stock unit balance and, if you had a discrepancy, it comes up with a message to warn you and say whether you want to post it to a local suspense. If you, at that point, hit the “Cancel” on the message you could then hit “Print” and carry on forward. It doesn’t rewarn you and it lost that discrepancy value.
So it produces a cash account – I’m sorry, a stock unit rollover that was out of balance so the payments didn’t match receipts. It was visible on the payroll, but it didn’t warn the postmaster again.
Then if they went to do the branch trading statement, when they roll the branch trading statement, they would get a non-zero trading position warning message because that stock unit had a payments/receipts mismatch.
Ms Patrick: So, really short, it showed an imbalance in the cash account ultimately, didn’t it?
John Simpkins: It showed an imbalance in the branch trading statement, yes.
Ms Patrick: Thank you. Now, if we can turn to page 2, please, we can see at the bottom of page 2 where the impact of this was analysed. And I’m going to look at the last three of those bullets, “Impact”:
“If widely known [this] could cause a loss of confidence in the Horizon System by branches.
“Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon data.”
Then, finally:
“It could provide branches ammunition to blame Horizon for future discrepancies.”
You can see that there on the record.
John Simpkins: I can.
Ms Patrick: So that’s discussing that the impact wasn’t simply on the inability of subpostmasters to reach a balance but there could be a wider impact because of the understanding of this problem being a system problem; is that fair?
John Simpkins: I think that’s fair.
Ms Patrick: Can we look at page 3, please, at the bottom. We have, I think, here – if I’m correct in my pagination – a list of possible solutions; is that right?
John Simpkins: Yes.
Ms Patrick: There’s 1, 2 and 3, and if we look at solution 2 there’s a number of suggestions there:
“P and BA will journal values from the discrepancy account into the customer account and recover/refund via normal processes. This will need to be supported by an approved POL communication. Unlike the branch ‘POLSAP’ remains in balance, albeit with an account discrepancy that should be cleared.”
I think that the recommendation you can see there is that that solution, solution 2, should be progressed; is that right? At the top, under “Proposal for affected branches”.
John Simpkins: Yes, the group’s recommendation is that solution 2 should be progressed.
Ms Patrick: Are you able to help us as to what happened: was solution 2 adopted?
John Simpkins: I’m not able to help you. I’m sure it should be relatively straightforward to find out.
Ms Patrick: Can we scroll to the top of that page, page 3. You can see there’s an introduction and a sort of overview explained there. At paragraph 2:
“Fujitsu are writing a code fix which …”
I think there is a “will” missing there:
“… which [will] stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week commencing 4th October. With live proving at the model office week commencing 11th October, with rollout to the network completed by 21st October. We have explored moving this forward and this is the earliest it can be released into live.”
So the problem was discovered in May, it’s brought to the attention of Post Office, I think, in September, and now the solution will not be actioned or live until October; is that correct?
John Simpkins: That sounds correct, yes.
Ms Patrick: Then it goes on:
“The code fix will …”
I think there’s another typo here:
“The code fix will on stop the issue occurring in the future but it will not fix any current mismatch at branch.”
Can you help us with what that would mean in practice?
John Simpkins: So if you have already got a payments/receipts mismatch in the stock unit and a non-zero branch trading statement, this fix won’t correct that, but it will stop it happening in future.
Ms Patrick: So there would be a problem that wouldn’t be fixed by the fix. Does that mean that something else would need to be done to address –
John Simpkins: The solution –
Ms Patrick: Apologies. Would something else need to be done to fix the mismatch that had already happened?
John Simpkins: You don’t have to do anything, apart from you – the imbalance would rollover – be brought forward and then be reported in the next branch trading statement and after that it would be cleared. So you don’t actually have to do anything as long as the Post Office is made aware of what has happened.
Ms Patrick: As long as the Post Office is aware that it has happened –
John Simpkins: Correct.
Ms Patrick: – and they are aware which branches may have been affected?
John Simpkins: Correct.
Ms Patrick: Thank you. I think we can move to the second document I would like to look at, Mr Simpkins. It’s FUJ00083770. It’s a series of emails. Can you see that in front of you now?
John Simpkins: Yes.
Ms Patrick: You can see, if we scroll to the very bottom, which I’m going to start with, where we see your name mentioned first – you can see there at the very bottom there was an email sent from Mike Stewart to you on 22 February 2006. Can you see that?
John Simpkins: Yes.
Ms Patrick: I’m not going to look at that yet, I’m going to scroll a little bit to the bottom, so that we can all see what the issue was. Can we go to page 6, please, at the bottom, a little further down, thank you. You will see there’s an email there from Shaun – it’s Shaun – it’s from Sandra McKay to Shaun Turner. Are you able to help us with who they were?
John Simpkins: No.
Ms Patrick: I think her title – somebody else might be able to help us. Sandra McKay, it says, is from sales and service, and it says:
“You may recall that in September the above office had major problems with their Horizon System relating to transfers between stock units.
“The subpostmaster has reported that he is again experiencing problems with transfers … which resulted in a loss of around £43,000 which has subsequently rectified itself. I know that the subpostmaster has reported this to Horizon support, who have come back to him stating that they cannot find any problem.
“Clearly the subpostmaster is concerned as we have just spent a number of months trying to sort out the first instance and he doesn’t want a repeat performance. He is convinced that there is something wrong with his Horizon kit. I would be grateful if you could investigate this and give him any support that you can.”
If we scroll back up a little bit, we will see the reply, or a further email in the chain, at the top of page 6. I think it’s a further email in the chain rather than a reply. It’s from Brian Trotter to Shaun Turner and do you know who Brian Trotter was?
John Simpkins: No.
Ms Patrick: So it says:
“Further to Sandra’s email, I visited the branch with Sandra last week and the subpostmaster provided clear documented evidence that something very wrong is occurring with some of the processors when carrying out transfers between stock units. To be absolutely sure from our side, can we either carry out a thorough check of the alleged faulty processors or swap them out.”
So from what we can see at this end of the problem, it’s the postmaster who has had an issue which has come back again and is being investigated, and somebody has also again witnessed that there is indeed an issue; is that fair?
John Simpkins: That’s fair.
Ms Patrick: Okay. If we can go back to page 1, please, and if we start at the bottom, with the email that Mike Stewart sent to you. If we scroll a little bit further down, please, we see there Mike Stewart is writing to you:
“John, did you get a chance to look at this? Do we think all will be well after S90 counter rollout?”
Was the S90 a new release –
John Simpkins: Yes.
Ms Patrick: – of Horizon Legacy?
John Simpkins: Yes.
Ms Patrick: If we go down, we can see – I won’t read it all out – in the first instance, he has tried to reach Anne Chambers but she was away; is that right?
John Simpkins: I presume so.
Ms Patrick: You see it says, “Anne is away, could I have your comments as you were involved as well”. He goes on to talk about the PinICL and he refers to there being a PinICL for this issue, and it says:
“The time out events are apparently fixed in a new Riposte version released at S90.”
There’s a PinICL number, and it says:
“I have looked at the problems and can’t see why the system reported disconnected nodes.”
He goes on a little bit, and he explains:
“I think the best thing now is to see what happens after S90. I’ll continue to keep this call open to remind me that this site should be checked then.”
It goes on a little bit to talk about the postmaster, or the person reporting the problem initially being reported as female, and it goes on again to say a little bit more about the problem. He refers to “a magical £43,000 appearing and disappearing” and the postmaster is then reported to be male, and he says, September:
“… the above office had major problems with their Horizon System relating to transfers between stock units.
“The subpostmaster has reported that he is again experiencing problems with transfers …”
It says:
“… which resulted in … around £43,000 which has subsequently rectified itself. I know that [he] has reported this to Horizon support who have come back to him stating that they cannot find any problem.”
It repeats almost the content of the email we have just discussed.
He goes on, he says:
“Sorry for this long-windedness. Is it a problem at the branch he wants to query? Is it Horizon kit or is there an issue with staff there?
“If there is an issue is this S90 release the cure? How confident are you/we it will fix the problem?”
Then he says the release is due in the week of 4 March. So we’re in February at this point, he is talking about a few weeks away; is that fair?
John Simpkins: Yes.
Ms Patrick: So he is posing some questions for you, originally for Anne Chambers to consider; is that right?
John Simpkins: Yes.
Ms Patrick: If we scroll up on page 1 we will see the reply doesn’t come from you, it comes from Anne Chambers and it says:
“I believe John has already responded to this, so don’t know if you need any more from me …”
It perhaps suggests that she has spoken to you before she has replied, doesn’t it?
John Simpkins: That seems reasonable.
Ms Patrick: It says:
“I haven’t looked at the recent evidence but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks, and finally Escher say they have done something about it. I am interested in whether they really have fixed it which is why I left the call open – to remind me to check over the whole estate once S90 is live – call me cynical but I do not just accept a third party’s word that they have fixed something!
“What I never got to the bottom of …”
She explains she is concerned why this particular branch had a particular problem. It goes on to say:
“… KELs tell SMC that they must contact sites and warn them of balancing problems if they notice the event storms caused by the held lock and advise them to reboot … before continuing with the balance.”
It says:
“Unfortunately in practice it seems to take SMC several hours to notice these storms by which time the damage may have been done.”
So it’s a problem there that we know has already been known about for years; is that right?
John Simpkins: The locking problem, yes.
Ms Patrick: There’s no solution as yet. They’re looking to S90; is that correct?
John Simpkins: Yes, so the locking problem stops counters communicating between each other, so it’s like having – what we were talking before about replication not happening and what had happened is with the (unclear) square they did a transfer out, they did the transfer in on one counter and then they could do the transfer in on another counter because it hadn’t got the transfer in messages replicated to it, so they had two transfer ins and one transfer out, so you had a payments and receipts mismatch.
Ms Patrick: So again a mismatch.
She says, Anne, that they still need to check the whole estate after S90 goes live?
John Simpkins: That’s looking for these events.
Ms Patrick: And there’s still a need to investigate further, isn’t there?
John Simpkins: Yes, so once the software has been rolled out, then you would need to check to ensure it has fixed the problem.
Ms Patrick: But she expresses the problem that:
“With this issue it can sometimes take several hours to detect the problem and by that point the damage has been done.”
John Simpkins: That was the workaround until the fix is put in place, so the workaround was that SMC monitor the events from the estate and the lock event – when they see the lock event, they contact the branch to reboot that counter, which was the workaround to fix the locking problem, so that then it will be replicating to the neighbouring counters.
Ms Patrick: If we go back to the email chain, the very last in the thread, we see a message from Mike Stewart to Anne Chambers which says:
“Anne, John did reply but just to say that Escher say they have fixed it? So, like you, we will have to wait and see what happens after S90 rollout.”
Again, what we’ve got there – there’s no solution as yet. Everybody is going to wait and see for S90; is that fair?
John Simpkins: That’s fair. The workaround is in place, but it is fair.
Ms Patrick: Again, the issue is that Escher say they have fixed it?
John Simpkins: Yes.
Ms Patrick: There’s no certainty at this point, is there?
John Simpkins: No, I think the third line support team are a cynical lot.
Ms Patrick: Indeed. Were Fujitsu here reliant on Escher for a solution?
John Simpkins: A fix to Riposte. This is a Riposte bug with the Riposte locking, so it wouldn’t replicate.
Ms Patrick: So for a Riposte bug –
John Simpkins: Yes.
Ms Patrick: – it needs an Escher fix?
John Simpkins: Escher write the Riposte software, yes, that’s correct.
Ms Patrick: You weren’t fixing it onsite, it had to be repaired by Escher?
John Simpkins: That’s correct. It’s a software – the new version of Riposte fixed that problem.
Ms Patrick: Thank you. I don’t think I have any further questions for you, Mr Simpkins.
Mr Beer: Sir, I don’t think there are any other questions from anyone.
Sir Wyn Williams: So that’s obviously a convenient time to break for lunch.
Mr Simpkins, thank you very much for providing your written statement and for answering questions during the course of the morning. I’m grateful.
John Simpkins: Thank you.
Mr Beer: Sir, before we break, there is a possibility that Mr Simpkins will come back to help us further in the Inquiry in later phases. I have been asked by the Fujitsu legal team whether they have permission to speak with him in the intervening period.
Sir Wyn Williams: Well, my short answer to that is yes.
Mr Beer: Yes. Thank you very much, sir. So did you say 2.00?
Sir Wyn Williams: Well, 5 past, I think.
Mr Beer: Did you? Thank you very much, sir.
(1.05 pm)
(The luncheon adjournment)
(2.04 pm)
Mr Stevens: Good afternoon, sir, can you see and hear me?
Sir Wyn Williams: Yes, I can.
Mr Stevens: Thank you. If I may call Mr Ascott.
Mark Ascott
MARK ASCOTT (sworn).
Questioned by Mr Stevens
Mr Stevens: Mr Ascott, as you know my name, is Sam Stevens and I ask questions on behalf of the Inquiry. Please could I ask you to state your full name?
Mark Ascott: Mark Andrew Ascott.
Mr Stevens: Thank you for giving evidence to the Inquiry today.
I want to start with the bundles in front of you. You should see a witness statement, and that is dated 9 August 2022, and at page 24 of that statement is that your signature?
Mark Ascott: Yes, it is.
Mr Stevens: Are the contents of that statement true to the best of your knowledge and belief?
Mark Ascott: To the best of my knowledge and belief, yes.
Mr Stevens: Thank you. That now stands as your evidence to the Inquiry but I will be asking you some questions, not on all parts of it and, in particular, I won’t be asking you questions today on the elements relating to Horizon Online, or at least not in any depth.
In your witness statement, you say that in August 1998 you worked for a Fujitsu business called The Solution Centre; is that right?
Mark Ascott: That’s correct.
Mr Stevens: At that stage could you summarise what qualifications you had relevant to IT?
Mark Ascott: So, as a member of The Solutions Centre I was assigned to various projects which would be involved with integrating, implementing and testing solutions for customers. At the end of August, I had completed three years on assignment to Barclays Investment Bank, where we had migrated them from 40 locations around Tower Hill and Royal Mint Court, down to Canary Wharf, so I had been involved in assisting a senior technical design authority in implementing NT domains and providing work stations and configuring work stations which were rolled out to the various BZW teams for their use in their new location.
Mr Stevens: Thank you. You say you were assigned to ICL Pathway from The Solution Centre between September and December 1998 to work on testing.
Mark Ascott: Mm-hm, that’s right.
Mr Stevens: What specifically were you testing at that point?
Mark Ascott: I was testing network access and the ability for the network to be robust to withstand sort of attacks, so man in the middle type attacks, those types of spoofing IP addresses, to spoof devices that shouldn’t be accessing – and those types of tests.
Mr Stevens: So you weren’t testing the EPOSS application?
Mark Ascott: No.
Mr Stevens: So, as you say, you transferred to the Pathway organisation, you say, in January 1999 and you worked on the secure builds development team, which we will discuss in due course, but that was under Alan D’Alvarez, was it?
Mark Ascott: That was under Alan D’Alvarez, yes.
Mr Stevens: So it was in the security domain?
Mark Ascott: That’s my belief, that Alan was looking after the security implementation.
Mr Stevens: You then say, in 2000, you moved to work for a team known as infrastructure products development unit. What role did that entail?
Mark Ascott: Well, initially, I continued with the secure builds, so I was on the secure buildings development team. I worked in the Bracknell office but I eventually was moved across to work in the Feltham office reporting into Pete Dreweatt and Ian Morrison.
Mr Stevens: Thank you. We will come to deal with that team in a moment. You left the Post Office account in August 2005 –
Mark Ascott: I did.
Mr Stevens: – and you returned, I think, in July 2008 –
Mark Ascott: That’s correct.
Mr Stevens: – and you went on to work on Horizon Online?
Mark Ascott: Yes, I did.
Mr Stevens: Just one point on that. In May 2009, there was an article in Computer Weekly by Rebecca Thomson, which criticised the Legacy Horizon IT system at that stage. Were you aware of that article at the time?
Mark Ascott: I don’t recall that specific article. I would have been aware that Computer Weekly and Computing would be writing articles but I wasn’t taking much notice of those.
Mr Stevens: It wasn’t sort of spoken about in the office, that a core product that your account was working on had been criticised by a trade journal?
Mark Ascott: I think there would have been discussion within the office but, from my part, I tend not to believe what journalists write. For me, when I started working for ICL in 1978, I read religiously Computing and Computing Weekly and – but, over time, I just believed that those publications were, you know, biased against ICL, so it wouldn’t have been a surprise to me that there would have been negative articles in those publications.
Mr Stevens: I want to move to your witness statement. If we could bring that up please, it’s WITN04760100 and paragraph 34, please. Thank you.
I apologise, I have the wrong reference there. What I will do is say this: in your statement, you refer to having – you say you recall that:
“High level test plans and test reports included Post Office staff as recipients and reviewers of draft and approved documents.”
Yes?
Mark Ascott: Yes.
Mr Stevens: Was that in relation to Legacy Horizon or Horizon Online?
Mark Ascott: It would have been Horizon Online.
Mr Stevens: In respect of Legacy Horizon, would that – that statement have held true?
Mark Ascott: I’m not sure on that. I wasn’t part of the testing team for – down at Feltham, that would have been testing with counters, so I am aware that Post Office staff did work closely with the SV&I team –
Mr Stevens: But you didn’t have direct –
Mark Ascott: But I didn’t have direct involvement in the production or necessarily the reviewing of those documents.
Mr Stevens: Does the same apply for design and development documents as well?
Mark Ascott: I would say the design documents around the security solution that I would have been involved in didn’t generally include Post Office recipients.
Mr Stevens: I want to then – in your statement you, under a heading, give quite a lot of detail on the testing that you say you were involved with as part of the Horizon IT system. Was that the testing you did for the network matters you discussed earlier?
Mark Ascott: No, the testing that I’m referring to in my witness statements are mainly based upon my experiences of Horizon Online.
Mr Stevens: Horizon Online, not Legacy Horizon?
Mark Ascott: Mm-hm. So I was aware of the testing team in Feltham where the system tests, the SV&I test rig, there was also a test rig that was dedicated to supporting performance testing and there was also a test rig that was used for release testing, so I met and worked with a number of the people in the test team, in my role as a developer, in supporting them to diagnose defects with the products, which myself or my team had caused them to have to find.
Mr Stevens: Could I just stop there. You referred to test rigs. Just in lay terms or for a non-expert, what exactly is a “test rig”?
Mark Ascott: My definition of a test rig is a combination of platforms and servers which are going to replicate the systems which will be used in the live service and, in relation to Legacy Horizon and Horizon Online, a combination with counters.
So some rigs would not necessarily have counters, you may create a – what we would refer to as a harness, where that just involves servers, such as, when I was looking after FTMS, we could have a test rig that just involved the local FTMS gateway and the remote FTMS gateway –
Mr Stevens: Sorry, just on that, can you explain what “FTMS” is?
Mark Ascott: FTMS was the file transfer managed service application used to move data from the database servers to other servers that was going to process that data.
Mr Stevens: So you have given us a definition of what the test rigs are and some of your involvement in it. Did you have any involvement in – or do you have any knowledge of how the process of balancing was tested?
Mark Ascott: No. I wasn’t involved in the sort of counter operations and the transactions and the processes during Horizon Legacy system, or even Horizon Online. I did have more involvement with counters during Horizon Online doing performance testing. That would have involved creating scripts which could be driven through a tool we used, LoadRunner, to create a load and those scripts would have embedded counter transactions.
And part of that would have involved processing of data created by the counter transactions within the data centre systems, primarily the database servers.
Mr Stevens: So, in essence, what we have here is a high level description of testing but your working knowledge of testing for Legacy Horizon is not strong?
Mark Ascott: Yes, so in the secure builds side, where I was working, I was generating scripts, which would translate the policy files and the requirements around the security needs that would result in the platforms being built in the test rigs, in a manner that was going to replicate the live service so …
Mr Stevens: Well, I do want to come to that shortly. Before moving on, you refer to something in paragraph 31 of your statement, “Maestro24x7 batch scheduler”. Was that a Legacy Horizon matter?
Mark Ascott: It was an application as part of the Legacy Horizon solution, which coordinated jobs that would be run in a 24-hour period in a sequence that enabled data to be harvested and collected and then processed by the various subsystems.
Mr Stevens: So you have already mentioned from testing that defects would be raised and they would need to be addressed by development teams, and that was through a system called PinICL.
Mark Ascott: It was.
Mr Stevens: That’s a repository to log these identified defects and maintain a central point of what has been done to investigate them and resolve them. Do you agree with that?
Mark Ascott: I agree that PinICL was that system that was used for defect management.
Mr Stevens: We have heard evidence about individual PinICLs being prioritised. Do you recall firstly who prioritised them in your team?
Mark Ascott: That could be a combination of people within the team. It could be the programmer that was responsible for that product, it could be myself and that programmer. It could be that we would seek guidance from the senior designers that were available to us, just to identify what the true impact would be.
Mr Stevens: How would you go about prioritising those PinICLs?
Mark Ascott: In terms of how quick we could fix them. One of the things which we learned over time was what business impact they would have but, initially, we were making assessments on “That’s a typo, we can correct a typo very quickly and easily”, or, you know, “That’s going to be a complex piece of work to fix”, and we would, generally, in those situations, turn to the designers to guide.
Mr Stevens: So, sorry, for the more complex fixes, would they be prioritised higher or lower?
Mark Ascott: They would likely be prioritised higher.
Mr Stevens: And the easier ones prioritised lower, but you weren’t assessing that on business need, initially?
Mark Ascott: Not initially. In the secure build space, where I was working, the defects that would come back to myself would usually prevent part of a test rig from being built or it may prevent a test rig from being built at all. So, clearly, if it was an issue where the test rig couldn’t be built, that was important and that would be prioritised highly.
If I had made a typo, a group had been named incorrectly, or a user account had been specified incorrectly, that would have been a lower priority defect and would have been fixed in accordance to the priorities that were set at the time.
Mr Stevens: Are you aware of anyone outside of ICL Pathway who had access to the PinICL database?
Mark Ascott: I’m not aware of anybody that wasn’t a member of Pathway having access to the PinICL system.
Mr Stevens: So the Inquiry has heard evidence this week concerning problems with EPOSS and that application, one of which was related to the malformed or incomplete messages created by Riposte. Were you aware of that at the time?
Mark Ascott: No, I wasn’t.
Mr Stevens: When did you become aware of issues with Riposte?
Mark Ascott: I think I would have been aware of Riposte issues around about 2001/2002.
Mr Stevens: Why was that?
Mark Ascott: Working more and more in Feltham, I had the wider Pathway design and development team around me, so I would have likely encountered people that would have been involved, designers that would have been involved in that counter solution. There would have been, like, chit-chat in the canteen, that type of thing.
Mr Stevens: So in around 2001/2002, is this roughly the point where you started attending the morning prayers meetings?
Mark Ascott: Certainly once I had taken on responsibility for managing the infrastructure teams, which I listed in my statement, there was a higher chance that, you know, one of those applications would require some representation at morning prayers.
Mr Stevens: Could you just – just so we can hear what your – why was it called “morning prayers”?
Mark Ascott: Simply because it was a very early morning meeting. I guess the reference goes back to, sort of, the religious context of, you know, morning matins and –
Mr Stevens: What did you do in the meetings?
Mark Ascott: I would attend with other attendees. I would listen to the issues that were being described there. Some may not involve me at all, but if there was an issue that was affecting the application set that I have listed in my witness statement, then the likelihood is I would be asked to take away and investigate what could be done to resolve a PinICL as quickly as possible.
Mr Stevens: Could you just take us through the application set in non-expert terms, as it were, to say what you were actually responsible for at these morning prayer meetings?
Mark Ascott: Yes, so secure builds, which was defining the NT domain structure, putting together the secure roles and secure role templates, defining the access control lists, which would be implemented against the NT platform set. FTMS, which I have described as the file transfer manager service, that application was used to move data, for example the TIP data from the database servers across to the Post Office systems. FTMS was also moved to move data to GiroBank for onward processing from GiroBank.
Audit-Dev, so there was the audit and archive solution which I was responsible for collecting data which would be stored for a period, which I knew to be seven years, and, you know, there was a set of definitions as to which data should be collected and recorded in that repository. Maestro-Dev, which we have touched on, so the driving part of the solution that kept the data flowing as and when it needed to be during the 24-hour cycle.
The other area that I was involved in was the auto configuration database, so that was the database that the counters would call into when they were being built and, on the counter, there was a utility called PC Config and Europa. Europa would do checks and trigger PC Config as needed for a new-build counter. PC Config would connect and then, if you like, the personalisation data would be provided by the Auto Config database to that counter.
Mr Stevens: So if and when there were discussions on problems with Riposte and/or the EPOSS application, would you be involved in those conversations other than as a listener?
Mark Ascott: I wouldn’t have been involved in those conversations other than as a listener.
Mr Stevens: At the morning prayer meetings, was this solely limited to people employed by ICL Pathway and later Fujitsu?
Mark Ascott: Yes. All the representation there would have been ICL or Fujitsu people.
Mr Stevens: You mentioned then that one of the areas over which you had responsibility was audit. Would you have been responsible for – or would that have included the audit of actions taken by support services such as the SSC?
Mark Ascott: I believe so.
Mr Stevens: Presumably, that also refers to audit for recording transactions that are carried out in Post Office Counters as well?
Mark Ascott: Yes, and the movement of data. So when – for example, FTMS would collect files from the database servers. That data would have been recorded and stored in the audit servers.
Mr Stevens: Was there a record of – an audit record of key strokes that a subpostmaster would use when using a Horizon terminal?
Mark Ascott: That I don’t know.
Mr Stevens: Could you tell us what the ARQ audit data contained?
Mark Ascott: No, I wouldn’t be able to tell you what data is recorded in it. I believe it would include a PAN number, a unique number associated with a debit card or credit card, but whatever data it would have recorded, I don’t know.
Mr Stevens: Can I then turn – I want to turn to the secure build Windows script and the secure build – I think “Secure Builds-Dev” it is referred to in the document, is that the team?
Mark Ascott: Yes, secure builds is what we were known as, the “Dev” was implied.
Mr Stevens: We have heard from Fujitsu employees this week evidence stating that it would be necessary for third line support to have write access to counter systems. Would you agree with that?
Mark Ascott: Yes, I would.
Mr Stevens: Subject, of course, to proper control over its use?
Mark Ascott: Yes, so the requirements for the secure roles were recorded in the document RS/Req/012, which I think Alan D’Alvarez authored, and that document there identifies the secure roles that needed to be created and listed, if you like, the attributes of each secure role, so …
Mr Stevens: Could we bring up FUJ00087994, please. Is this the document you’re referring to?
Mark Ascott: Yes.
Mr Stevens: As you say, it is drafted by Alan D’Alvarez. Did you have any input into it?
Mark Ascott: I think I would have had discussions with Alan to sort of, if you like, shape what that group definitions would look like.
Mr Stevens: We don’t need to turn it up but, in your witness statement, you say, at paragraph 20:
“I regarded the security architects I worked with as subject matter experts and deferred to their knowledge and design thinking. If I did not understand part of their designs, I would discuss my concerns with them so that I could gain a complete understanding of their designs. Once the NT domain designs were approved by the senior design team in Fujitsu and were due to be taken forward, I developed tooling and scripts that were used to implement the secure builds on the various Windows platforms.”
Mark Ascott: Yes.
Mr Stevens: So is this saying that you would use this document and, when you say the security architects, is that referring to Alan D’Alvarez?
Mark Ascott: No, I’m referring to Belinda Fairthorne there, so Belinda was ICL’s senior security architect. Certainly in 1990, I was part of an organisation of which Belinda was a key member of the senior architects group and we reported into ICL’s chief technology officer at the time.
Mr Stevens: She was the person who wrote the access control policy?
Mark Ascott: She – she certainly contributed to that production of that document, yes.
Mr Stevens: When it came to designing the scripts to make, as you say, the secure NT build, would this have been the primary document you would have used?
Mark Ascott: This would have been one of the documents that I used. I had meetings with Belinda and Barry Procter to go through a set of rules and requirements which would then provide the framework that would enable me to shape the NT domain design.
So that NT domain design was the start point for the work that I did with Belinda. From the NT domain design document, which in this group definitions for secure NT build document that Alan authored, in the tables that are included in this document, the columns include the domain names, and Alan would have needed those domain names in order to fit the group definitions for the secure roles to build that picture up.
Mr Stevens: Well, let’s look at one of those tables. If we could, please, turn to page 9, thank you. So it says “Group Name to be implemented”, on the left, and “SSC Apps MAN”, that’s SSC application management?
Mark Ascott: Yes.
Mr Stevens: You were referring to the domains earlier. Which part of this table were you referring to?
Mark Ascott: So, in this table here, the authentication domain column lists the authentication domains where users accessing those domains are going to be verified in terms of user and passwords.
Mr Stevens: So could you assist us in decoding what’s under the “Authentication Domain” column?
Mark Ascott: So the “PWYDCS”, that domain name refers to Pathway data centre systems.
Then the “TEWKDLR”, “SITTDRL” and “DUNSDLR” and “WYCODLR” refer to domains associated with De La Rue, so there would be FTMS servers located within those four sub-domains that were used to transfer data between Pathway systems and De La Rue systems.
Mr Stevens: So, for present purposes, really, we’re only concerned with the Pathway one at the top. Where it says “NT, All Servers”, when you reviewed this document what did you think that covered in terms of scope?
Mark Ascott: That would be all NT platforms that were members of any Windows NT domain.
Mr Stevens: So it would include post office counters?
Mark Ascott: No, counters were excluded. These are Windows NT platforms that are located in the data centres, Wigan and Bootle.
Mr Stevens: So Mr D’Alvarez in evidence yesterday said that it would include counters. You disagree with that?
Mark Ascott: Yes, I disagree with that.
Mr Stevens: In the – well, we can go there, I think. Bear with me, sorry.
Actually, I think it will be hard to bring the document up, but in the security functional specification, which was shown yesterday, one of the points that was made was that in Riposte, a Windows NT workstation was described as a message server, Riposte message server. Is your evidence that that – because of that, this table – it doesn’t mean a counter falls within this table “All servers”?
Mark Ascott: That’s right. The counters were excluded from the Windows NT domain design.
Mr Stevens: But this does include the correspondence server?
Mark Ascott: The correspondence servers and the agent servers would have been catered for within this design.
Mr Stevens: Do you have any recollection as to what the Tivoli remote console did?
Mark Ascott: My recollection of the Tivoli remote console was that it was – that, if you like – the device that was used to apply changes to the data centre systems.
Mr Stevens: To the data centre systems, was that, sorry?
Mark Ascott: Yes.
Mr Stevens: It’s right, isn’t it, that it would also be used to make changes to counters?
Mark Ascott: I have no recollection of it being used for counters, but my focus and my work was concentrated on delivering for the data centres.
Mr Stevens: So it wasn’t within your remit to look at how the SSC may or may not be able to access counter computers?
Mark Ascott: Well, I translated the document that Alan has authored here, which is shown on the screen, so the definition of the group name “SSC Apps MAN”, I took that information and I created Windows NT scripts that would enable that tool set that’s listed in the second column on the left under “Tools”, so that those tools could be made available to the secure role, SSC Apps MAN, and I would have also created the access control lists, which would have enabled rewrite and execute to the folders on the various platforms described under “NT servers”.
Mr Stevens: Do you know who was responsible for access control as between the SSC, or anyone in Pathway for that matter, and the counter branch computers themselves?
Mark Ascott: No. No, I’m not aware of the security fit that was applied to counters.
Mr Stevens: Given you were based in the security team and you were involved in access control in some way, does that – can we infer that no one was overseeing that aspect, namely who had access between the SSC and the counter systems?
Mark Ascott: I have no knowledge of anybody that was applying the same controls that I was applying to the Windows NT servers that were located in the data centres.
Mr Stevens: Please could I go to FUJ00088036. This is a document that’s been shown a few times this week. It’s the “Secure Support System Outline Design” dated 2 August 2002. Do you recall if you saw this document in 2002?
Mark Ascott: I’m not listed as a reviewer of that document, but I do recall Geoffrey Vane. I do recall working with Geoffrey Vane.
Mr Stevens: Sorry, Geoffrey?
Mark Ascott: Geoffrey Vane.
Mr Stevens: Why is that relevant to this document, sorry?
Mark Ascott: I believe that Geoffrey Vane, as the security TDA at the time, would have been involved in this document.
Mr Stevens: If we could turn the page just to see the distribution list. So yes, Geoffrey Vane, security TDA. Do you recall being – well, let’s first go to page 9, if I may, so we can set out what the document did. So SFS, do you know what that refers to?
Mark Ascott: I believe that refers to the security functional specification.
Mr Stevens: So the security functional specification:
“… mandates the use of Tivoli Remote Console … for the remote administration of Data Centre platforms. This records an auditable trail of log-ins to all boxes accessed by the user. It is a matter of considerable discussion and correspondence that TRC is slow and difficult to administer. This has led over time to BOC personnel relying heavily on the use of unauthorised tools (predominantly Rclient) to remotely administer the live estate.”
With that context, do you recall having discussions about this document, or the issues raised in it in 2002?
Mark Ascott: I don’t recall the discussions in terms of the detail that’s included within this document, but I do remember the PWYSAS domain being created and the –
Mr Stevens: Sorry, could you just explain what that is, sorry?
Mark Ascott: So that would be Pathway secure access server, or servers, and then within that sub-domain I remember the MBOSAS01(?) and MWESAS01(?) so those would be been the servers that were hosting the terminal services.
Mr Stevens: What was the purpose –
Mark Ascott: (Unclear) designed the first two.
Mr Stevens: So those domains were created to essentially give effect to the tool proposed here?
Mark Ascott: Yes.
Mr Stevens: And you were involved in the creation of those?
Mark Ascott: I believe so, yes.
Mr Stevens: Could we please turn to page 19. If you scroll down a little, this diagram – is that explaining the tool that is being proposed to allow audited access to the counter systems?
Mark Ascott: Yes. I think that diagram there is describing what we knew as the Cygwin solution.
Mr Stevens: And could you just explain what that solution entailed?
Mark Ascott: There was a terminal server, as described on the right-hand side. On the secure access server there was an SSH client and a similar SSH component logged on –
Mr Stevens: What’s the SSH client?
Mark Ascott: It was a third party solution that enabled secure connection between a client and a server and it enabled certain functions, such as the capture of user authentication and the auditing of that log on.
Mr Stevens: So was this to be used by members of the support services essentially to remotely access counters?
Mark Ascott: I believe so, yes.
Mr Stevens: And you say it had – well, we can see from the document that there were enhanced, or more audit features available, is that correct?
Mark Ascott: Yes.
Mr Stevens: Do you know what data was captured for audit purposes?
Mark Ascott: No.
Mr Stevens: Was audit not within your domain?
Mark Ascott: I didn’t need to know the detail of data records that were being captured. What I needed to do in that role was to make sure that the development team, looking after and making changes to the audit solution, were aware of those changes that they needed to code and my role was to report to people like Pete Dreweatt and Ian Morrison whether the audit team was on track or not.
Mr Stevens: So if we can go further down on this document please there should be a small – I think it’s over the next page, sorry. Thank you. This is referring to the secure support access server, SSAS.
Mark Ascott: Yes.
Mr Stevens: Can you just assist with what that is?
Mark Ascott: Well, that is the secure access server that I was referring to earlier.
Mr Stevens: That’s what you were referring to?
Mark Ascott: Yes, yes.
Mr Stevens: And the additional security domain, it says:
“Third line and Operational Support Units have, and require, system admin access to all systems they manage. In order to create a non-refutable audit of all actions carried out against systems they manage it is necessary to restrict their access to the system which is gathering the audit.”
So in essence is that saying support services shouldn’t be able to access the audit data captured through this tool?
Mark Ascott: Yes.
Mr Stevens: It goes on – we don’t need to read it all – to refer to a second security domain being required and changes will be required to the existing security NT domain hierarchy. In the box it says:
“Question for Mark Ascott: Could this be as simple as moving the administration of the PWYDCS user domain to a separate group?”
Could you please just elaborate on what that question meant?
Mark Ascott: I think that question is exploring, you know, is there a simple change to the NT domain structure, or, you know, is there a more involved change. The person asking that question, whoever wrote that design note, is clearly looking to me to provide some guidance as to how simple or quickly the change can be made, or, you know, is there more complexity required.
Mr Stevens: And do you recall when this change was made?
Mark Ascott: I think this is part of the conversations and the work that I would have done with Geoffrey Vane in order to bring this in, this change in. As I say, for me my recollection is that I recommended that we introduced the PWYSAS domain and placed the secure access servers as member servers of that specific domain, that sub-domain.
Mr Stevens: When this tool was developed, presumably it required testing?
Mark Ascott: Yes, it would have done.
Mr Stevens: Were you involved in that side of things?
Mark Ascott: I would have been involved in delivering the changes that introduced the PWYSAS domain and setting up the secure role users and the secure profiles, the access controllers. My scripts would have been passed to the PIT team, as it was known then, the product integration team, and the first real test for those scripts would have been when the SPTS team, the service provision and test services group – they were a small team, they were located in Feltham, and they would have used those scripts after the PIT team had wrapped around a deployable script in to enable the software deployment tools to execute those scripts. So they would have been involved in the testing for real, in terms of did those scripts enable the PWYSAS domain to be added to the data centre solution.
Mr Stevens: But do you have any recollection of the results of the testing related to this tool?
Mark Ascott: No, no. So in my own testing of my own scripts I would have desk checked them. I wouldn’t have had available to me an equivalent test rig of all of the NT systems and all of the NT domains, so I would have desk checked and I think there was a technique that I used to use where I could execute the scripts but using Echo to print, if you like, the commands that were being executed and I would have received those responses back and I would have then been able to tell whether the logic in the script was doing what I intended it to do.
Mr Stevens: But that was all prior to going for formal testing?
Mark Ascott: That’s right, so that was my own development testing that I was doing to satisfy myself that the changes were ready to be made available to the product integration team.
Mr Stevens: And as part of your role in developing this tool and as part of your role in audit, did you have any oversight as to whether or not this tool was being used for the purpose of remotely accessing counters?
Mark Ascott: So I’m providing the basis for the platform to be part of the overall domain structure. The application – the SSH application that’s sitting on top of that is a tool that I’m enabling the secure role and the users that are created from that secure role to execute. The configuration of the SSH tool itself, I was not delivering that configuration.
Mr Stevens: I want to ask a few points about something that was raised in evidence with Graham Allen yesterday. Did you listen to his evidence?
Mark Ascott: I did see some of Graham’s recording, yes.
Mr Stevens: He said that – I’m paraphrasing, but that the team developing Horizon Online was not able to look at the code for Legacy Horizon for intellectual property reasons, is that right?
Mark Ascott: Yes. I remember in 2008 when I rejoined Pathway in the test team that there was a lot of emphasis put on the counter team not reviewing or going anywhere near what was the Riposte solution for fear of claims that, you know, the new HNG-X counter that was being developed was a copy, or was, you know, a facsimile of the Riposte system and that was a message that was regularly mentioned, certainly in 2008. You know, although the counter development team was located in the Braro 1 Tower(?) and I was primarily working in the test team lab area on the ground floor, the ripples of those types of reinforcing that message would reach down into the test teams.
Mr Stevens: Are you aware of whether that inability to look at the Legacy Horizon code caused any difficulty, or any – it held things up when developing Horizon Online?
Mark Ascott: Again, I wasn’t part of the counter development team, so I can’t really say how they went about solving the problems of delivering and developing the new HNG-X counter. You know, we were very much aware that, you know, there was this – it didn’t feel like a directive, you know, “You will not go and look at that previous counter design or that previous counter code”, but it wasn’t far away from that. The direction was relatively clear that, you know, we could – well, ICL/Fujitsu could be taken to task for copying Riposte, so …
Mr Stevens: In your role and when working with Legacy Horizon did you ever have to deal with third party software suppliers?
Mark Ascott: Can you repeat the question again please?
Mr Stevens: Yes, of course. Let me put it another way. I know you didn’t work with Riposte, but that was an application provided by Escher –
Mark Ascott: It was.
Mr Stevens: – a third party, and the question I ask is: in your roles, did you have something similar where you had to work with a third party outside of Pathway in respect of an application you oversaw within Legacy Horizon?
Mark Ascott: Well, I would say no to that. I mean as part of developing the secure role templates, those templates needed to refer to applications and executable files from a range of applications, but, you know, what I was processing was, you know, “This is the tool, this is the executable – this is where you can expect that to be located on the NT server or the NT work station”, so if I knew the location of the executable I could configure that into the secure role template so –
Mr Stevens: So no equivalent then to working with Escher?
Mark Ascott: I didn’t have to have any dealings with any third party suppliers. In Horizon Online obviously, as part of the volume and integrity test activity, we utilised the Hewlett Packard LoadRunner tool set and in order to use that, so tools and applications, we needed to have access to their support desk, you know, when we encountered issues with using that tool, but that was solely for Horizon Online and the HNG-X solution.
Mr Stevens: Sir, I wonder if that might be an appropriate time to have a break?
Sir Wyn Williams: Yes, it would. Sorry about the slight delay but I was on mute.
Mr Stevens: Thank you, sir.
Sir Wyn Williams: So what time shall we start?
Mr Stevens: Quarter past, would that work?
Sir Wyn Williams: That’s fine.
Mr Stevens: Thank you.
(3.03 pm)
(Short Break)
(3.14 pm)
Mr Stevens: Good afternoon, sir, can you see and hear me?
Sir Wyn Williams: Yes, I can. Thank you.
Mr Stevens: Mr Ascott, I only have one question really. You said at the beginning of your evidence, towards the start, that when I referred to the 2009 Computer Weekly article, you thought that publications or journalists were biased against ICL, yes?
Mark Ascott: Yes.
Mr Stevens: Do you still hold that view, or have you changed your mind knowing what you know now about Legacy Horizon?
Mark Ascott: What I know now from news media organisations like the BBC, ITV, Sky, I would say yes obviously I am aware that prosecutions were made that not necessarily should have been, but at that time I had had an experience whereby a journalist wrote for our local newspaper that a teenage girl who had been glue sniffing –
Mr Stevens: Sorry, is this relevant to –
Mark Ascott: Well, it sets my mindset in relation to articles produced and written by journalists, so for me, you know, my experience in that instance … I take with a pinch of salt what journalists write.
Mr Stevens: But, as I say, knowing what you know now, what is your view of the Computer Weekly article?
Mark Ascott: Yes, knowing from the court cases and the judgments that have been passed, now I know differently.
Mr Stevens: Sir, I have no further questions, but I understand that there are – I will just check if there are any questions in the room, sir.
(Pause)
Mr Stevens: No, it’s a nil return, sir.
Sir Wyn Williams: There we are, Mr Ascott. The questions that have already been put to you are the ones that people want to put, so thank you very much for coming to give your evidence and for answering the questions and thank you for making your written statement as well.
So I take it that concludes this afternoon’s business, Mr Stevens?
Mr Stevens: That does, sir. We will be back at 10 o’clock tomorrow if that suits you.
Sir Wyn Williams: Yes, of course. So 10 o’clock tomorrow morning.
Mr Stevens: Thank you, sir.
(3.17 pm)
(The Inquiry adjourned until 10.00 am on Thursday, 10 November 2022)