10 November 2022 – Mark Jarosz and Peter Jeram
Hide video Show video
(9.59 am)
Mr Blake: Good morning, sir.
Sir Wyn Williams: Good morning.
Mr Blake: Today’s first witness is Mr Jarosz.
Mark Jarosz
MARK JAROSZ (sworn).
Questioned by Mr Blake
Mr Blake: Good morning, could you give your full name, please?
Mark Jarosz: Mark Jarosz.
Mr Blake: Mr Jarosz, you should have in front of you a bundle containing a witness statement. Is that statement dated 9 August of this year?
Mark Jarosz: Yes, I have.
Mr Blake: Yes. Could I ask you to turn to the final page, page 21 of 22, and is that your signature there?
Mark Jarosz: Yes, it is.
Mr Blake: Thank you. Is that statement true to the best of your knowledge and belief?
Mark Jarosz: Yes, it is.
Mr Blake: Thank you, Mr Jarosz. I’m going to ask questions on behalf of the Inquiry today. Thank you very much for attending today and thank you for your witness statement. The witness statement, for the purposes of the transcript, is WITN04810100. That statement and the exhibits will all go into evidence and what I’m going to ask you today will build upon what’s already in there. I think we will probably be half a day, possibly less.
I’m going to start with your background. You joined ICL in 1983; is that right?
Mark Jarosz: Yes, it is.
Mr Blake: You became employed by ICL Pathway or what became ICL Pathway?
Mark Jarosz: Yes.
Mr Blake: You started as a customer support executive?
Mark Jarosz: I did, yes.
Mr Blake: Then you became involved in Horizon from 1995 to 2012; is that right?
Mark Jarosz: Yes.
Mr Blake: So you were involved before even ICL had succeeded in the procurement exercise?
Mark Jarosz: Yes, I was.
Mr Blake: In 1996, you became a solution architect networking; is that right?
Mark Jarosz: Yes, it is. That was my role, yes.
Mr Blake: That was your role, and that was Legacy Horizon, or what’s now known as Legacy Horizon?
Mark Jarosz: Yes.
Mr Blake: Then in 2010 to 2012 you were a solution architect security, was that in relation to Horizon Online?
Mark Jarosz: Yes, it was.
Mr Blake: You are still employed by Fujitsu; is that right?
Mark Jarosz: I am, yes.
Mr Blake: Your title now is lead domain architect?
Mark Jarosz: That’s correct.
Mr Blake: Presumably, you still have access, therefore, to Fujitsu records and things like that?
Mark Jarosz: Yes, partially I do, yes.
Mr Blake: You are represented today and assisted by the Fujitsu legal team?
Mark Jarosz: Yes, I am.
Mr Blake: Can we bring up on screen your witness statement. It’s WITN04810100, please. Now, this statement was provided in response to a Rule 9 request to Fujitsu on 11 March of this year for a corporate statement relating to phase 2 of the Inquiry. Are you aware of that?
Mark Jarosz: I am, yes.
Mr Blake: You were chosen by Fujitsu to be one of several witnesses who we have heard from to respond to that request.
Mark Jarosz: Yes.
Mr Blake: Now, in that original Rule 9 request, there was a section about robustness; do you remember that?
Mark Jarosz: I do, yes.
Mr Blake: Yes, and it asked for an explanation as to what was known by ICL about the accuracy and integrity of the data recorded and processed on the Horizon System.
Mark Jarosz: Yes.
Mr Blake: It also asked about the extent to which deficiencies in the Horizon IT system were capable of causing or caused apparent discrepancies or shortfalls; do you remember that?
Mark Jarosz: I do.
Mr Blake: After you gave a draft statement, you were sent a second request asking for more detail in certain respects; do you remember that?
Mark Jarosz: I do.
Mr Blake: That request was 1 July of this year. Again, in that request there was a broad question about robustness. Do you remember that?
Mark Jarosz: I do.
Mr Blake: You may not remember but, in that request, at the top, there was a section in capital letters saying that you are expected to have refreshed your memory from contemporaneous documents. Do you remember that section?
Mark Jarosz: I do.
Mr Blake: Let’s look at your statement. It begins with an introduction. Could we scroll on to the next page please. It then has a background section and, over the page, it goes on to talk about the bid for the Horizon project and, over the page, this is paragraph 12, and there you say that there were a number of decisions before you came onto the scene, one of which was the ISDN decision, to use ISDN, and the second was to use Riposte. Is that a fair summary of that paragraph?
Mark Jarosz: Yes, it is.
Mr Blake: They are two big decisions that are mentioned in that paragraph. You have said a number of decisions but, presumably, you see those as the two significant decisions that were taken before your time?
Mark Jarosz: Yes, those were the two main ones. There are a few further ones as well.
Mr Blake: The point that you make is that the decision to choose Riposte was not your decision.
Mark Jarosz: That’s correct.
Mr Blake: Your initial role, I think, was to do performance modelling on Riposte; is that right?
Mark Jarosz: Yes, on Riposte and the network, yes.
Mr Blake: Can we look at paragraph 18, so we can scroll on a little bit more. Thank you very much. At paragraph 18, you say:
“At this initial stage, I did have some concerns about whether the Riposte messaging solution would effectively scale to approximately 20,000 branches, as it had not been proven to work at that scale before. This was not a concern that was unique to me, but was a known issue that was actively discussed within the bid team and with Escher.”
Looking at paragraph 19, you say – I will just turn to my own copy:
“Managing the issue of scaling Riposte was not within my … responsibility. However, I do recall, from my general involvement on the architecture team, that this concern was eventually addressed in the deployment phase (during and prior to the pilots and rollout of Horizon).”
So again, what you are making clear there is that that wasn’t your responsibility, the scaling of Riposte, but it was addressed?
Mark Jarosz: Yes. I was very much aware of that.
Mr Blake: Can we look at paragraph 21, please. In that paragraph, you set out the approach that had been taken to Riposte and how it had been decided that it would operate. So, again, it’s emphasising there that that wasn’t your decision as to how to operate Riposte; is that correct?
Mark Jarosz: That’s correct.
Mr Blake: Paragraph 22, please. You say there you didn’t have any concerns about the use of Riposte in that manner. So, again, it wasn’t your decision how to use it but you didn’t have any concerns about its use in the manner in which it was used; is that correct?
Mark Jarosz: That’s correct.
Mr Blake: Can we look at paragraph 24 and 25, please. 24:
“In order for this design to function on the Horizon System, Escher needed to develop new software for use on Riposte.”
So 24 and 25, I think, explain the new software that needed to be developed and then, scrolling over to 26, it says there that you worked on the ISDN network solution, so that was the focus of your work there; is that right?
Mark Jarosz: Yes, that’s correct.
Mr Blake: Can we look over the page to paragraph 27, and you say, in respect of the ISDN work that you carried out:
“… the bid team internally convinced ourselves that the ISDN solution was sufficient.”
Mark Jarosz: Yes.
Mr Blake: So –
Mark Jarosz: Sorry.
Mr Blake: Sorry.
Mark Jarosz: Yes, that’s correct. It took a while to come to that conclusion.
Mr Blake: Yes, so that’s the area that you say you were responsible for, the ISDN connection, and you were ultimately convinced that it was sufficient; is that right?
Mark Jarosz: Yes.
Mr Blake: Paragraph 29, please, and onwards address the Initial Go Live pilot. I think you highlight in that paragraph, or in paragraph 31, that the Initial Go Live was limited from your perspective because it had a permanent ISDN connection, so it didn’t test the more intermittent ISDN connection.
Mark Jarosz: Yes.
Mr Blake: But 32, so scrolling down, you didn’t recollect any specific problems that arose during that Initial Go Live phase; is that right?
Mark Jarosz: Yes, not within my area, which was the network area.
Mr Blake: Yes. Over the page, to the 200 to 300 branch pilots. Again, you say there, in paragraph 34, you don’t recall any problems occurring; is that right?
Mark Jarosz: Yes, that’s correct, in my area, which was the network at that stage.
Mr Blake: Then 35 onwards addresses the pilot and the rollout of New Release 2. At paragraph 38, please, you observe:
“During the pilot, we observed a number of issues as we worked towards scaling the Horizon solution”, and you set out there three issues.
I think (a) could be summarised as moving some external storage; is that right?
Mark Jarosz: Yes, it is.
Mr Blake: (b) is providing a VSAT to remote branches, so instead of the ISDN certain branches could use a satellite connection?
Mark Jarosz: Yes, so that was dealing with the fact that ISDN, although it was the primary network technology, wasn’t available everywhere, so there needed to be an alternative solution.
Mr Blake: And (c), if we could keep on scrolling to (c), software updates needed to be scheduled differently because they were all taking place at the same time and causing some difficulties; is that right?
Mark Jarosz: Yes.
Mr Blake: Then we go to paragraph 40, please, where you say:
“Beyond the points above, I do not recall the issues that arose during the NR2 pilot. However, I believe they were … typical of [any] large-scale IT projects of the time.”
You don’t recall any particular issues that contributed to the delay of the NR2 pilot or the rollout of the system.
It is paragraph 46 then that addresses the issue of robustness and I’m going to read that paragraph. It says:
“I am aware of the Inquiry’s definition of ‘robustness’. I am only able to evaluate the Horizon system’s robustness from the perspective of my roles on networking and security, and I note that I had a much more limited involvement in relation to Horizon Online than its predecessor.”
Just to be clear, there is a section in your statement on Horizon Online that I have skipped over.
Mark Jarosz: Yes.
Mr Blake: “It was also not my role to design or develop the applications that would have recorded/processed data on Horizon, including in relation to branch accounts. From that perspective, I did not have concerns about the robustness of Horizon, nor was I aware of any.”
Can I just clarify, was there another Mark Jarosz working at ICL in 2000/2001. It’s a pretty unique name, presumably you were the only Mark Jarosz?
Mark Jarosz: Only one, yes.
Mr Blake: You have been given some papers over the past few days, many of which with your name on, which relate to Riposte bugs, what’s known as “Riposte lock” – commonly referred to as “Riposte lock”, and that is known to have fed into what we know as the Callendar Square bug. Which paragraph of your statement do we find mention of the Riposte lock issues?
Mark Jarosz: So in terms of the Riposte lock issues, the reason I was involved in that was because the people working on the problem needed to find out from Escher what the error messages meant and, at the time, there were a very few of us who had a working relationship with Escher. So my role was to ask questions directly, face-to-face with Andrew Sutherland from Escher about what that meant and convey his response to the people working on the problem in ICL at the time.
Mr Blake: Yes, and where in your statement can we find reference to the Riposte lock problem with Horizon?
Mark Jarosz: I didn’t mention the Riposte lock problem in my statement.
Mr Blake: Did you follow the Group Litigation, the Bates and Others case, did you follow that at all?
Mark Jarosz: In the press as it was reported, yes.
Mr Blake: So you still work for Fujitsu, so presumably it’s quite well-known?
Mark Jarosz: Yes.
Mr Blake: Did you, presumably, understand the significance of those Riposte lock events in the context of that case?
Mark Jarosz: No, sorry, I didn’t.
Mr Blake: Did you follow the Callendar Square incident at all?
Mark Jarosz: No, sorry, I didn’t.
Mr Blake: I’m going to take you to the documents in a moment but it looks from those documents that you were quite a central figure in trying to resolve or deal with Escher in relation to that Riposte lock problem. Is that a fair description of your role?
Mark Jarosz: Well, I was working with Escher at the time on the networking aspects of Riposte, which meant I spent time in their facilities in Boston, USA, and when people working on such issues had questions of them then, because there wasn’t much documentation, to the best of my knowledge, about the Riposte – the messaging product, the way the questions were resolved was to ask them directly, face-to-face and, whilst it was the case that, during the bid phase, Escher did attend ICL offices in Feltham, at that stage, they were mainly in Cambridge, Massachusetts. So my role was to convey those questions directly to Escher and get responses and feed those back.
Mr Blake: So you were being given problems by engineers working on particular problems and your role was the direct liaison with Escher in relation to those problems?
Mark Jarosz: Yes. There were other people, not just me, involved in the liaison but not many and I was one of them.
Mr Blake: Yes. I mean, it’s fair to say from that that you were fairly involved in trying to resolve Escher-related bugs in that case, weren’t you?
Mark Jarosz: Well, as one of the examples shows, my role was to convey the information back to our teams so they could progress with what they were doing. In many cases, the information I provided was not sufficient for them to resolve the bug but allow them to progress with it.
Mr Blake: So is your evidence that you were simply the liaison with Escher –
Mark Jarosz: In that particular example of –
Mr Blake: – and you weren’t making decisions – I mean, similar to the other parts of your evidence, where you say “Decisions were taken and I was simply following them”; is that the position in relation to Riposte lock?
Mark Jarosz: In the example that you gave, Riposte lock, that was the case. There are other examples which were also in the pack, where I was asked by the architecture group to take a more proactive role.
Mr Blake: But in Riposte lock you didn’t take a proactive role?
Mark Jarosz: No.
Mr Blake: And there are other bugs that you did take a proactive role in relation to?
Mark Jarosz: Yes.
Mr Blake: Where are those mentioned in your witness statement?
Mark Jarosz: So the example was a Riposte bug and I didn’t mention it in my witness statement. This is – I think it is E1, it was called the “handle leak problem”.
Mr Blake: We will look at the handle leak problem as a background. Can I just ask while we are on this issue – we can take down the witness statement, thank you – what was your relationship with Gareth Jenkins at this particular time?
Mark Jarosz: So I would describe it as professional, based on the need to work together, because we were part of the – at the time, Alan Ward’s team, so Gareth would – when Gareth was aware, for example, that I was going to visit Escher, he may ask me some questions to convey to them.
Mr Blake: Were you senior to him; at the same level?
Mark Jarosz: Same level. We worked in – we had different responsibilities within the architecture team, but we were level.
Mr Blake: We will go to the correspondence in due course, but it looks, from some of that correspondence, that he is looking to you for guidance; would you accept that?
Mark Jarosz: No, because he was a peer working at a different part of the solution. So whilst I was responsible for the networking part of the solution, he was responsible for the counter and agent applications.
Mr Blake: Would you say you had joint responsibility then for certain issues?
Mark Jarosz: Well, I can imagine that could arise, yes, where there was an issue where it wasn’t clear where the issue lay.
Mr Blake: I mean, something like the Riposte lock problem, would you have joint responsibility for that?
Mark Jarosz: Well, no, because, in that particular example, what Gareth wanted to know from Escher was what that error message meant. The Riposte product logged lots of error messages and there was no documentation which said what this error message means and what the consequences could be, so he needed someone to ask that question and, in some cases, he asked me; in other cases he would have asked the liaison that was at Escher, because we had people who were there on secondment to act in that liaison role.
Mr Blake: So, again, you were the conduit rather than the person who was responsible?
Mark Jarosz: Yes, one of them, yes.
Mr Blake: Were you ever asked to give statements in criminal proceedings?
Mark Jarosz: No.
Mr Blake: Were you ever involved in who would give such a statement?
Mark Jarosz: No.
Mr Blake: As peers, why was Gareth Jenkins selected and you weren’t; do you know?
Mark Jarosz: I don’t have knowledge of why that was.
Mr Blake: Were you ever involved in researching historic issues with Riposte, more recently, for example?
Mark Jarosz: No.
Mr Blake: I’m going to take you to a document, it’s POL00028911. This is a document that we may well come back to and I don’t think it’s necessarily a document you have seen. Is it a document that you are familiar with at all?
Mark Jarosz: No, I don’t recognise that document.
Mr Blake: So the only relevance, for current purposes, are that it concerns the Callendar Square bug and, if you look at the list of PEAKs, it lists the PEAKs that are related to that issue, and one of them is PC0056922, and that’s something that we’re going to come back to in due course. So we can take that document down for now, but we will look at that particular PEAK.
Let’s look at the contemporaneous documents from 2000/2001. Can we look at FUJ00078274, please. So this is going to be a bit of background before we get to the particular PEAK. This is an ICL “Weekly Progress Report” for 30 July 2000 to 2 August 2000. Can we look at page 3, please.
So this is a document you are familiar with and I think you have already referred to one of the issues that’s raised there and let’s have a look at those. Can we scroll down that page, please – a little bit more, so that we have the whole of that 1.2 in view, please?
So here there are two major critical issues arising during the week. The first, handles leaks in the Riposte message server which could ultimately threaten rollout if not resolved and it says “An urgent fix is being sought from Escher”. That’s the one you referred to just a moment ago, is it?
Mark Jarosz: Yes, it is.
Mr Blake: Again, that one isn’t mentioned in your statement, is it?
Mark Jarosz: No, it isn’t, but –
Mr Blake: Can you very briefly explain what that relates to, the leaks in the Riposte message server?
Mark Jarosz: Yes, so during – I believe during testing, it was observed that some resources used by the Riposte message server were increasing and the testers were concerned that that behaviour suggested there was a leak in the Riposte message server.
Mr Blake: What does a leak – what does that mean?
Mark Jarosz: It means that it’s using resources in a manner that eventually it will run out of resources and stop working. So that was the interim conclusion reached by tests and, therefore, it raised quite a few concerns. So my role was to ask – initially – this was agreed within the architecture team – was to describe the scenario to Escher and ask them whether this was a bug or behaviour as designed.
They confirmed it was – Andrew Sutherland confirmed this was behaviour as designed, so within the architecture group we then decided to see – and, by the way, Andrew Sutherland also explained to me why this was happening and when it would stop.
Mr Blake: Can I just ask, who is Andrew Sutherland?
Mark Jarosz: He is the chief architect for the Escher group messaging product. So he is the kind of person who knows about the product the most.
Mr Blake: Would he be your direct liaison with the Escher group?
Mark Jarosz: Yes.
Mr Blake: There’s a second problem that’s mentioned there. The second problem is the failure to swap out slave counters on – we have seen this before, is it “CI4”? Is that something you remember, or is it “Cl4”, “C14”?
Mark Jarosz: I think it is “CI4” but I just – I remember it as being one of the releases that we were doing.
Mr Blake: Yes, and it says:
“At present, intermittent fault causes the Riposte service to hang.”
It continues:
“Investigations of slave swaps has shown the problem occurring at a number of different points in the process of copying the squirelled message store”, et cetera.
Can you briefly explain what that issue was at all?
Mark Jarosz: No, I wasn’t involved in that, so I – I wasn’t asked to help with that issue.
Mr Blake: Is this a document that you would have seen at the time though, ICL weekly progress report?
Mark Jarosz: Well, I may have received it on an email but I can’t remember reading it.
Mr Blake: I mean, do you remember receiving Pathway weekly progress reports in 2000?
Mark Jarosz: I do recall being copied on them, yes.
Mr Blake: Would it not have been of interest to you?
Mark Jarosz: So, yes, I would be interested, if there were network issues, and in the issue that – the handle leak issue, it was called to my attention, so I was involved in dealing with it.
Mr Blake: Are you able to assist us with what it means by “intermittent fault causes the Riposte service to hang”? Is that a lock issue or is that something else?
Mark Jarosz: I can speculate what that means, in general terms, because, if Riposte is hanging, I would assume it means it is unresponsive and can’t be used for anything and needs to be restarted.
Mr Blake: Can we look at page 6, please. At the bottom, there is a section on “Current Critical Problems”, and there are the two problems there that we have just discussed. The first is getting the squirelled message store, they can’t successfully swap out a faulty counter on CI4, and then the second one is the issue “in live with handle leak”, and it says there:
“Gareth Jenkins will address this issue. In the meanwhile Mark Jarosz will liaise with Escher to establish the root cause of the leak.”
Mark Jarosz: Yes, so just to confirm, that’s exactly what I did: I liaised with Escher and I fed back my findings to the team internally within ICL. As a result of that, because Riposte was working as designed, based on the feedback, the decision was made to attempt to reproduce the problem, or reproduce the scenario, in our test facilities in Bracknell where we had the ability to simulate thousands of counters connecting to correspondence servers, and that proved that this was not an issue.
Mr Blake: Now, as I said, this is – I’m taking you to this for background and to establish the roles and responsibilities.
Mark Jarosz: Yes.
Mr Blake: It seems as though Gareth Jenkins and yourself are the prime, principal contacts with regards to Riposte errors, at that stage; is that right?
Mark Jarosz: So Gareth Jenkins’ role in this was based on the assumption that this is an issue that needs to be addressed and how we would mitigate that in the live solution. My role with the performance team was to find out if we needed a fix from Escher or whether this was working as designed.
Mr Blake: I mean, what you’re doing: you’re not just kind of passing messages to Escher though, are you? You’re described here as establishing the root cause of the leak, or working with Escher to establish the root cause.
Mark Jarosz: Yes, but, in this particular example, it – a very brief conversation with Andrew Sutherland confirmed that there was no problem, so the assertion there was a leak was incorrect and, in order to test that, we – that’s why we ran it on this test facility we had in Bracknell to confirm all was – there was no problem.
Mr Blake: I’m not concerned with the particular issue that occurred here. I’m more concerned about the different roles and responsibilities.
Mark Jarosz: Okay.
Mr Blake: Certainly reading here, you are acting as more than just simply a messenger with Escher; you are the person who is liaising with them, in order to find out the root cause of the problem?
Mark Jarosz: That’s very true, yes.
Mr Blake: Was that typical of your job?
Mark Jarosz: I can only recall a few issues that I was asked to look at, which are of this significance to the programme, and this is one of them. So, no, it wasn’t typical. My normal day job was the evolution of the network, which also included changes to Riposte to work over the network.
Mr Blake: Would it be typical for Gareth Jenkins to be working on the technical side of something and for him to ask you to liaise with Escher to try and resolve it?
Mark Jarosz: Well, there are examples where he has done that, yes, but typically by email, but the – what he asked me to do was to ask specific information of Escher and, typically, that would have been there’s some observations made based on error messages and what do they mean, if that was not already known to him.
Mr Blake: Typically to establish the root cause of a problem?
Mark Jarosz: Yes, partly problem investigation.
Mr Blake: Can we look at FUJ00083544, please. Thank you very much.
Now, this is the PinICL that I mentioned earlier and that was mentioned in that Callendar Square document. The PinICL itself is at the bottom, it has been forwarded, and it is PinICL 56922. Can you see that? The title, in the subject at the bottom?
Mark Jarosz: I can, yes.
Mr Blake: Thank you. Can we go over the page to page 2, please. I’m going to take some time over this document. Can we scroll down slightly on this page. There is an entry at 19.15 on 1 November. Yes, it’s the fourth entry there, and it says:
“PM [that’s postmaster] reports error message when trying to redeclare her cash.”
Thank you. It says – there’s another entry there:
“Guided caller thru redeclaration:
“STK …”
Do you understand what it’s saying there, just that entry “STK bal/dec cash …”
“Dec” may be December, perhaps? I don’t know, it may not be.
Mark Jarosz: I’m not 100 per cent sure what the abbreviations mean, whether it’s referring to the navigation on this counter, I …
Mr Blake: “Error message says ‘error committing declarations’
“Voiced call to Dave in smc …”
“SMC”?
Mark Jarosz: I think that’s one of our support teams.
Mr Blake: Yes:
“… who requested I pass the call over to them. Caller [advised] and ref [number] given.”
Then it says:
“User ‘ADA001’ advises that when a SU (CASH) declaration is made the declaration would not be accepted – “searched kel for Error committing” – nothing.
“Searched events from web PAGE for counter 1 – ‘An unexpected error occurred [while] attempting to modify an entry in the run map. Timeout occurred waiting for lock’ and also critical ‘Error Number …’”
It gives an error number, et cetera:
“The Riposte PutObject function call returned an error – this happened while”, et cetera, et cetera.
Then we go down the page and it shows that at 22.16, so that’s near the bottom of the page:
“Repeat Call: [postmaster] is still waiting for a phone call it has been three hours since this issue arose. Please ring immediately.
“The [postmaster] is only still available due to living on the property.”
Can we go over the page, please. The first substantive entry there is 2 November still, 9.24:
“as pm [postmaster] is trying to redeclare cash to alter she is getting error in declaration of cash declaration error in committing list.
“Pm tried to create a new declaration for the difference and got the same message.”
Do you understand that at all?
Mark Jarosz: Well, in general terms, I understand that these are operations being performed on the counter, yes.
Mr Blake: Is this an example – I don’t know – of a postmaster trying to re-enter a declaration because of the problem they are experiencing?
Mark Jarosz: It is hard for me to say because I’m not familiar with the counter application and how it’s used.
Mr Blake: Okay. Let’s move down, please, and it says there – it is the entry about halfway down the page, or three-quarters of the way down:
“The above kel outlines the problem …
“HSH1 Information:
“Called [postmaster] on the [advice] of Sara in smc to get the messages [postmaster] is getting, [postmaster] would like call back as is now trading manually and is not being called back to get problem solved.”
So it looks as though the postmaster there has stopped using Horizon and is trading manually. Do you agree with that interpretation?
Mark Jarosz: Yes.
Mr Blake: Then slightly below, 9.38, if we could scroll down a little bit, it says:
“The call summary has … changed from:
“PM reports error message when trying to redeclare.”
It is now:
“… error committing declarations.”
Is that something you understand at all?
Mark Jarosz: No, I’m sorry, I don’t.
Mr Blake: Could we go over the page, please. There’s an entry at 9.40 on the next page, and it says there:
“This call has been raised to ‘A’ as [Post Office] is manual due to being unable to roll over SU due to events being generated by gateway which SSC are actioning as per KEL.”
It has effectively been given an “A” priority:
“Mike Woolgar rang in. I explained situation and he requested that he be paged again if situation not resolved by 13.00.”
Can we go down to 10.30, please. It seems there:
“nbsc chasing …”
It’s a priority call:
“nbsc say [postmaster] is on manual, [postmaster] was called this morning by 2nd line and told nonsense. [Postmaster] is very angry and feels that she is being messed about. Contacted edsc who states that haven’t called pm. Called smc is checking with the person who was dealing whether they called [postmaster] will call back. Nbsc says will call back in 20 minutes if no resolution.”
Were you, at that time, familiar with these kinds of concerns from postmasters?
Mark Jarosz: No.
Mr Blake: 10.36, the entry there says:
“If nbsc ring back on this call please contact an stsa. Has given a 20 minute deadline in which she is calling us back.”
10.46, slightly further down the page:
“Spoke to Les – passing call over urgently. Advised user to reboot as she was stuck in a loop … and contact NBSC as to extending [Cash Accounting Period]. Message store and Event log audit logs coming.”
Now, were you aware, or are you now aware that a workaround in relation to this problem was rebooting?
Mark Jarosz: Well, I’m now aware that’s been mentioned, but the –
Mr Blake: Do you remember your state of knowledge about the Riposte lock issue and whether a workaround was, at that time, to reboot?
Mark Jarosz: No, and that wasn’t the advice that was given, that I recall from Andrew Sutherland either.
Mr Blake: But you would accept that that is the advice that’s being given in this particular PinICL, “Advised user to reboot as she was stuck in a loop”?
Mark Jarosz: Yes, I mean, it’s very clear, yes.
Mr Blake: Can we go over the page, please, and it’s about halfway down the page, 11.22. It says:
“The call record has been transferred to the team: EPOSS-FP.”
Who were EPOSS-FP?
Mark Jarosz: I’m sorry, I don’t know who that team are. I’m not sure what “FP” stands for.
Mr Blake: If we go down to the entry after, so 11.48:
“The Call record has been transferred to the Team: EPOSS-Dev.”
Is that your team?
Mark Jarosz: No.
Mr Blake: What team is that?
Mark Jarosz: Well, given that EPOSS is a counter – well, is an application, I guess it’s an applications team that look after – there were many applications in Horizon, and EPOSS was one of them, so I would assume it’s the team who looked after the EPOSS application.
Mr Blake: Could we go to the next page, please, page 6. There’s an entry by Martin McConnell. Who was Martin McConnell?
Mark Jarosz: I don’t recognise that name.
Mr Blake: He says:
“In my first analysis of the message store supplied, it would appear that the declarations being written away were done so at the time that the EOD process kicked in. The message which indicates the Riposte failure …”
It says there “putpersistentobject”:
“… should have allowed the user at least to have backed out and start again, which seems to happen satisfactorily when these conditions are simulated on a development system. As Les has indicated earlier, a system restart should be sufficient to get them back and working.
“OK, in which case I would suspect this call should be dropped to a ‘B’. Will see if I can simulate the failure whilst in the midst of an EOD scenario.”
So is Mr McConnell there – is a fair interpretation of that that he is going to try and simulate what the problem was. Is that a typical response?
Mark Jarosz: Yes, that’s my reading of it.
Mr Blake: We see there there’s a customer call again:
“Paged Mike again as per his last request as gone 3 pm and call still not resolved. Awaiting his call back to advise.”
Customer call:
“Mike called to advise that if call not resolved by [6.00 pm] then to page the Duty Manager again.
“Call updated as requested.”
Then it’s the next entry that is really the significant entry on this PinICL that I want to ask you about. Mr McConnell says:
“I have talked to Brian Orzel …”
Who is Brian Orzel?
Mark Jarosz: Brian Orzel was one of our developers and he is also the person who spent quite a bit of time in Escher facilities in the States in a tactical liaison role as well.
Mr Blake: Spoken to him “about the ‘lock’ errors written away by Riposte and it would appear that this is an indication of Riposte being rather sick.”
Is that a technical term? What would you understand by “sick”?
Mark Jarosz: I’m not sure how to interpret that. There’s many possible interpretations.
Mr Blake: “There are several DIIs …”
What are DIIs?
Mark Jarosz: I think that’s referring to DLLs.
Mr Blake: DLLs?
Mark Jarosz: So where that – so “DLL” and “executable” are computer code.
Mr Blake: So:
“There are several DLLs and executables all being told to go away because of this locking problem. Either some application has left some write lock on inadvertently or Riposte is sick as described.”
Again, “sick”, does that assist you at all?
Mark Jarosz: Again, it’s hard for me to interpret what that means, but …
Mr Blake: “A reboot should sort this out or try redeclaring on an alternative system. Brian Orzel has suggested routing this for the attention of Mark Jarosz.”
What do you have to say about the suggestion that it should be for your attention to deal with that issue?
Mark Jarosz: So I assume from that that Brian wants me to find out from Escher what the right course of action is for this particular error message. What I can’t tell from the date was whether Brian was already out there or not, onsite with Escher.
Mr Blake: Can we look at the first page of this document, please. At the bottom of the first page this PinICL seems to have been sent to Gareth Jenkins on 3 November. What was Gareth Jenkins’ role here?
Mark Jarosz: So within the team, Gareth was the Riposte technical design authority.
Mr Blake: If we look at the top email, please, Gareth Jenkins is emailing you, presumably following up from Mr Orzel’s comment, and he says there:
“I don’t know if you have been phoned about this one. It seems to have been passed to you on the Escher-dev stack.”
What was the Escher-Dev stack?
Mark Jarosz: So within PinICL, there’s multiple groupings for different people and I think Escher-Dev is one of those groupings.
Mr Blake: It refers there to what the problem is, including the message:
“Timeout occurred waiting for lock.”
He says:
“I assume the problem is down to the previous Query from EPOSS, however I can’t see why that would cause a one-off problem on this system.
“I don’t know if it is relevant, but the machine appears to have been rebooted in the middle of the night a couple of days earlier (ie at 02.00 and twice at 03.00 on [30 October]). The counter appears to be at CI4 …”
Now, we mentioned that earlier. We have previously in this Inquiry seen an email to Gareth Jenkins, where Gareth Jenkins is copied in, about CI4 and that email expressed concerns regarding counter performance and code regression with CI4. Is that something you remember at all?
Mark Jarosz: No.
Mr Blake: What is Gareth Jenkins asking you to do here?
Mark Jarosz: So I – well, I think the first thing that he is asking is for confirming with Escher, if this has not already been done previously, what “error 82” means and what the consequences are.
Mr Blake: Presumably you would have read the PinICL that was forwarded to you. So, at the bottom of this email, he is forwarding the full message to you. Would you have read that at the time?
Mark Jarosz: I would expect to, yes. I can’t remember that particular email but, in general, yes.
Mr Blake: I mean, those comments about Riposte being “rather sick”, that message went to you at least, didn’t it?
Mark Jarosz: Yes, it did.
Mr Blake: We started today, the first document we looked at, or the second document we looked at was about problems earlier that year with Riposte and you mentioned one of them was resolved but there were two critical issues with Riposte that were mentioned in that earlier document that I took you to. Was this building on your knowledge of issues with Riposte at all?
Mark Jarosz: So I think the first part of the question is about the error message and what I cannot recollect is whether I have asked this question of Escher before or not, or whether it had to be asked for the first time, about what that error message actually means. So I think that’s certainly one thing that’s being asked in the email.
Mr Blake: Would you have been concerned to have received a PinICL that said that Riposte was sick?
Mark Jarosz: Well, in general, yes, and it – I think the PinICL – in general with problems like this, unless the error message explains the problem, there is a need to reproduce the problem. So if that’s, indeed, what happened, then that would be the right course of action.
Mr Blake: Was it something that you think should have had Escher’s urgent attention?
Mark Jarosz: Yes, most definitely, based on the priority, yes.
Mr Blake: Can we look at FUJ00083548, please. Now, on the second page we see the PinICL, it starts on the very bottom of the first page, but it’s the second page and it’s a PinICL that is from 9 November, so just a week later. The reference here for this PinICL is PC0057478, and we see on the second page, about halfway down, the entry at 21.55, it says a critical error was registered:
“An error occurred while attempting to destroy a checkpoint run. Timeout occurred waiting for lock … no suitable kel.”
Are you able to help us with that at all? It’s not listed on that document that I showed you – the first document that I showed you to identify the relevant PinICLs or PEAKs for the Callendar Square problem, but is that also a Riposte lock issue that’s being reported there?
Mark Jarosz: Yes, it is. So the – this is another example where there’s an error message reported by Riposte and the – whilst I don’t recollect this particular example, what I would have done, in general, is I would have taken this to Escher and asked them for feedback about what the error means, what the consequences are on the message store and what the right course of action would be.
Mr Blake: Would you have taken them to Escher on every occasion?
Mark Jarosz: Only when asked because I wasn’t the only person who was liaising with Escher. So, if I was asked, either by email or verbally, to follow up, then I would do that. I would take the opportunity whilst I was out there to do that.
Mr Blake: So every occasion you were asked, you would go to Escher and try and resolve the issue?
Mark Jarosz: Well, I would certainly take the issue to Escher and feed back on the question I was asked. It wasn’t always possible in a timely manner because, sometimes when I was working there, the people who I needed to ask weren’t there.
Mr Blake: Can we look at the first page, please. If we look at the top – well, at the bottom it seems, again, to be a PinICL that went to Gareth Jenkins, on 20 November in this case. He emails you at the top on 21 November. They are American date formats but I’m confident that that is 21 November. Why would Gareth Jenkins have emailed you on this occasion?
Mark Jarosz: Because he wants Escher to confirm details of what “error 94” means.
Mr Blake: Can you just have a look at this document and tell us in simple terms what’s going on.
Mark Jarosz: So in the third paragraph, starting “However I am curious”, he is asking – he is quoting some error messages that were logged by Riposte and he is then stating he assumes they are benign “but would appreciate confirmation from [myself] before closing the PinICL”, and the only way I can seek that confirmation is by asking Escher.
Mr Blake: Assuming it is “benign”, that’s something we will see again, is that an assumption that something is going to be okay but it’s not a definitive position?
Mark Jarosz: Well, the – it’s probably building on – so understanding what the error message means is part of analysing the possible problem it could cause and I think only on conclusion – once analysis is complete, it could be concluded, maybe, that these messages can be ignored. However, I would say, in general, that if it is an error message it does need to be analysed.
Mr Blake: So again, it’s a PinICL, the detail of which is being sent to you by Gareth Jenkins for you to take up with Riposte, is it?
Mark Jarosz: So my response to this email would be to ask Escher for details of the error message, under what circumstances it occurs and what the consequences are, and then feed that back to Gareth, either verbally, face-to-face or via email, whichever.
Mr Blake: You would do that in every case when you are asked to?
Mark Jarosz: Well, where it’s a very specific question, “What is this error message?” yes, I would, but if I was unable to have that conversation with Andrew Sutherland then it may be quite a few weeks before there’s any response.
Mr Blake: That final substantive paragraph talks about ClearDesk. Now, I think ClearDesk was a way of resolving this Riposte lock issue because ClearDesk, I think, effectively restarted the system; do you remember that at all?
Mark Jarosz: I recognise the term “ClearDesk”, but I wasn’t really aware of the counter architecture and what processes ran when on the counter.
Mr Blake: Gareth Jenkins says there:
“Each time it is put out as part of the ClearDesk close down function. ClearDesk continues OK, so again it isn’t serious, but we need to avoid any errors being generated at the counter as part of ClearDesk (since they cost Pathway 3p each for a phone call!).”
Can you tell us about that, please?
Mark Jarosz: Yes, so, at this time, the networking was the ISDN dial-on-demand network and what that meant was that there was no connection between the counter and data centre normally but, when there was a need for communication, this ISDN phone call would be established. And what Gareth is asserting there is that if a – one of the conditions for actually forwarding messages to the data centre – in this case, it was a Tivoli function – if there’s a red – an error event logged by anything on the counter, then Tivoli will forward that to the data centre for investigation and that is the phone call that’s being referred to.
Mr Blake: So is that him saying “We would rather not spend the money on the phone calls”?
Mark Jarosz: Well, given that there were quite a lot of phone calls going on anyway, I’m not sure he is directly concerned about the cost of a phone call because – I mean, what I would say is that there may well be other reasons why there’s a call made anyway at that time.
Mr Blake: If we look at the very final sentence there he says:
“… assign the PinICL to me on Escher-Dev until I get feedback from you both.”
I asked about Escher-Dev before, does this assist your memory, is Gareth Jenkins part of that Escher-Dev team?
Mark Jarosz: So within PinICL, there’s multiple groups and, by implication, Gareth is part of that Escher-Dev team because he’s just – what you said, yes.
Mr Blake: Can we look at FUJ00083568, please. This is an email to you a few days later, 24 December 2000, and can we look over the page, please, page 2 – in fact, actually I think we can stay with page 1.
The PinICL there, the reference is PC0057957 and that is dated 16 November but it relates to the first PinICL that I took you to, ending 56922, and it says that at the very top of the page. It says “This PinICL is related to” that PinICL, which is the one that’s later linked to Callendar Square.
Can we look, please, over the page to page 2. Again, it refers to a critical event was registered and it says:
“Timeout occurred waiting for lock.”
So, again, that seems as though it’s one of those Riposte lock issues.
Mark Jarosz: Yes.
Mr Blake: Can we go to page 3, please. 23 November, 11.10:
“This event was reported in PC0056922, this call has been closed but the comments from Mark Jarosz, were that if calls of this nature were [over] 1 per month then further investigation should be carried out. In this case I presume that archiving was processing and there was still an outstanding lock on the run table. I presume that the reload of Riposte at ClearDesk will release the locks. Investigating frequency of event in the estate.”
Now, the suggestion there is that it wasn’t on every occasion that you were asked that you would investigate, you would apply some sort of minimum threshold of a problem before going to Riposte.
Mark Jarosz: So in the example of the error message, then it’s very clear that, because within Pathway we didn’t know what – we had no documentation to tell us what the error message meant, we had to ask Escher what it meant.
Mr Blake: But if you received a one-off incident, or what you considered to be a one-off incident, would you go to Escher?
Mark Jarosz: If Gareth asked me to, yes, I would.
Mr Blake: So the suggestion there that really they need to be looking out for a more common occurrence, where would they’ve got that idea from?
Mark Jarosz: So I think in one of the examples where we discussed reproducing the problem, then the – that’s what we talked about, the frequency of it occurring.
Mr Blake: We will talk about reproducing in a moment because it seems as though you had concerns that, if it something couldn’t be reproduced, there wasn’t really any point in going to Escher; is that right?
Mark Jarosz: Well, it’s more a case of if we need to go to Escher because we have found a bug in Riposte – and this occurs – this is a more general statement. If we need to investigate a bug then we are very keen to reproduce it so we can then both investigate it with a vendor and also confirm the fixes worked.
Mr Blake: Sticking with this particular issue, can we go down slightly to the next substantive entry. It says:
“This event has some 129 counters reporting this and also MBOCOR02 and MBOCOR03 has reported this event although it may be expected on the Corr servers.”
So is that correspondence servers?
Mark Jarosz: I think it is, yes.
Mr Blake: “I think this needs investigating. Please state what evidence is required will attach Event log/message store & audit logs for this outlet.”
Then if we go down a little further it says that it is 13.17:
“The Call record has been assigned to the Team Member: Gareth Jenkins.”
Then if we look at the first page, it is Gareth Jenkins emailing you. So, again, he has emailed you with the full detail of that PinICL. Would you have read that PinICL at the time?
Mark Jarosz: I can’t recall reading that particular one, but I, in general, would try to keep up with the emails, yes.
Mr Blake: So the message, for example, that the event has some 129 counters reporting, that was sent to you and typically you would read those messages that were sent to you?
Mark Jarosz: Yes.
Mr Blake: Now, it says at the bottom there:
“I’ve assigned the PinICL to you on Escher-Dev.”
Again, so does that assist you with Escher-Dev?
Mark Jarosz: I’m aware what Escher-Dev is –
Mr Blake: Yes.
Mark Jarosz: – (unclear), yes.
Mr Blake: So you were being assigned because you were part of that team?
Mark Jarosz: Well, I think it was assigned to me because, in terms of it – the next step in that PinICL, what Gareth was asking for was a definitive statement from Drew on that error message. So the next stage in the workflow for that PinICL would be to update it with that statement.
Mr Blake: And Drew is –
Mark Jarosz: So Andrew Sutherland, he is the chief architect from Escher group, the expert on Riposte messaging product.
Mr Blake: Okay. Mr Jenkins says:
“As the PinICL says, this seems to be happening fairly frequently. As far as I can tell, the application is carrying on OK in this case. Since the failure is at midnight, then Riposte is likely to be reloaded fairly soon.
“I think we do need a definitive statement from Drew as to whether this event is benign, or what problems we could have when it happens. Could it be due to an application error? Do we need to get more info on when these problems occur. It is clear that the circumstances in this case are very different from those in the original PinICL.”
Now, Mr Jenkins there seems to be concerned about repeated errors and where they come from; do you agree with that?
Mark Jarosz: Most definitely, yes.
Mr Blake: He says there he doesn’t seem sure that it’s benign, by that stage.
Mark Jarosz: Well, until we get – we need the feedback from Escher to explain the error message, which I think we actually got maybe in this example. I don’t know if there’s an email from me with a feedback from Escher.
Mr Blake: Well, we will come to an email from you.
Mark Jarosz: Okay, okay.
Mr Blake: You are being sent that by Gareth Jenkins, again, to take forward with Escher, to take forward with Drew, to see if it’s benign or not?
Mark Jarosz: Yes, although I wouldn’t actually ask Drew if it’s benign or not, just ask him to explain it and what the consequences are.
Mr Blake: Can we look at FUJ00083574, please. This is an email from you to Gareth Jenkins. It is about the same PinICL ending 957, and you say there:
“Gareth,
“From your description it sounds as though we potentially have a recipe for a reproducible case.
“I will try this today and also in parallel chase Drew for a response on what this event means and whether we should be concerned.”
The reference there to “reproducible case” –
Mark Jarosz: Yes.
Mr Blake: – again, I think we discussed this briefly, but it’s something that does crop up from time to time, and it looks like what is being said is that, without a reproducible case, it’s difficult to progress the problem. Is that something you agree with or not?
Mark Jarosz: Yes, it’s much more difficult to progress a problem that we can’t reproduce, yes, unless it’s a previously known issue.
Mr Blake: It looks from this and other correspondence that you do, at least, apply some criteria in respect of following things up with Escher. In fact, in this case, you say you are going to, but if it wasn’t a reproducible case, if it seemed like a one-off issue, would you always send it to Escher?
Mark Jarosz: Yes, most definitely. The reason for mentioning the potentially reproducible case, is that it makes the interaction with Escher potentially much more productive because, as well as asking them what could happen, we can actually demonstrate what is happening.
Mr Blake: Can we go to FUJ00083582, please? This is now 1 December 2000, and this is – is this an update to Gareth Jenkins on this issue?
Mark Jarosz: So this – in this case, I have responded to his question about the particular error message and what the feedback I had from Drew was, as –
Mr Blake: Sorry. You say there:
“Hi Gareth,
“I can confirm (having checked with Drew) that a timeout of this sort is likely to be benign in the sense that it should not result in a message store corruption.
“However had the operation which was affected by this timeout been a message server internal operation, for example and index maintenance thread operation, then an additional error … should have been logged.
“Therefore a possibility is that an API call has timed out and the application is not checking for error events.”
Now, that update: likely to be benign, “should not result”, possibly an API call has timed out, et cetera; would you accept that those are quite caveated responses?
Mark Jarosz: Yes, they are, based on conversations with Escher and the limited information we have available, trying to say what could be happening. For example, Escher are making the point that, if something was affected in the message server, there would have been further error messages and, as it’s their product, they can say that’s by design. So, even though I used the term “should have been logged”, I maybe should have used the term “highly likely” that it would have been logged, because Escher said so.
Mr Blake: But it looks from that message that you haven’t got to the bottom of the problem?
Mark Jarosz: That is definitely the case, yes, because the next part is very significant and this, again, is based on conversations with Escher, that because there’s an error message and something has timed out, then something was trying to happen, and if it wasn’t an internal message server operation, because Escher said so, then the suggestion is that the – and we know there was an agent because Gareth mentioned this running at the time, then the agent may have caused – an agent operation may have caused the error, which is why the suggestion from Escher was “Check that the agent is validating all responses from interactions with the Riposte message server”.
Mr Blake: I will come to that (a) and (b) in a second, but the words, for example, “mostly benign” or “relatively benign” are words that we have seen elsewhere and we may see in further emails, and again “likely”, “should”, et cetera. Does that indicate, perhaps, that you couldn’t be sure that there wouldn’t be serious problems arising from this Riposte lock issue at that stage?
Mark Jarosz: So I’m definitely not sure that is the case and there is further investigation needed, yes.
Mr Blake: Then looking at those (a) and (b):
“In terms of progressing … I would suggest …”
So this is you suggesting, not just simply passing a message, but you are coming up with your own solution?
Mark Jarosz: No, that was not the case. When I discussed this with Drew and we made the observation that there were no other error messages from the message server, he stated that, as there was an agent running, then the agent possibly would have had error responses, which should have been logged and possibly they weren’t, which is why the recommendation for (a) is directly as a consequence of what Drew asked me to do: is the agent checking all the responses correctly?
Mr Blake: But you weren’t simply passing a message, you were applying your own mind to the issue as well, weren’t you?
Mark Jarosz: So in the case of (a), it was Drew’s recommendation to check that the application – the agent or the application using the Riposte message server was checking its responses correctly.
Mr Blake: No, I mean, let’s look at it. (a) says:
“Get the LFS Agent code checked to confirm that all API calls have error checking. I am happy to do this if the developers are prepared to send me the source.”
Now, we have heard about issues accessing Escher code. Is that referring to an issue accessing the original code?
Mark Jarosz: No, this is – the LFS agent code is our code, so the –
Mr Blake: So who were the developers that you are talking about there?
Mark Jarosz: Well, the development team that created that agent. I don’t know who the individuals are.
Mr Blake: So you’re saying there that you’re happy to assist if the ICL Pathway developers are prepared to send you the code?
Mark Jarosz: Yes, I mean, what I should have said – it’s a bit tongue in cheek. I should have said “They should check it themselves”, because they should be checking all replies.
Mr Blake: Were there issues internally with getting hold of source code?
Mark Jarosz: No, the – because these were internal people doing development, they would have the source code for their own agent. So they could get this checked relatively easily.
Mr Blake: “(b) Continue to try and reproduce this problem. Knowing what the Agent is doing (either source code or some design documentation) would be useful.”
So it seems there that a solution is to keep on trying to reproduce it. So, at that stage, it seems it hasn’t been reproduced or is not yet reproducible?
Mark Jarosz: Yes.
Mr Blake: Just looking at the time, sir, shall we take a short break now? I probably only have 20 minutes left and then there will be some questions from recognised legal representatives. We can either take a break now or in 20 minutes.
Sir Wyn Williams: No, let’s do it now because I think we have to think of the transcriber as well. So let’s do it now.
During this short break, Mr Jarosz, would you please not speak about your evidence with anyone. All right?
Mark Jarosz: Okay.
Sir Wyn Williams: Thank you.
Mr Blake: Shall we say 25 past or half past?
Sir Wyn Williams: If the people in the room think that we will complete the witness comfortably before lunch, I’m happy with half past.
Mr Blake: Yes, I think the answer is, yes.
Sir Wyn Williams: Right, half past then.
(11.16 am)
(Short Break)
(11.30 am)
Mr Blake: Hello, Chair, we can see you. Can you see and hear us?
Sir Wyn Williams: Yes, I can, thank you.
Mr Blake: Thank you very much.
Before the break, the first documents I took you to we saw early Riposte issues in the summer of 2000 and then I took you to some PinICLs that were later in 2000, that addressed things such as the Riposte lock.
Can I take you to POL00028911, please. Now, if you would like to take some time over this document, please do. It’s a document that we’re not too sure where it’s come from. It may have been written by Gareth Jenkins, that may be established in due course. It’s a document that I took you to first today, and I showed you the PC0056922 being referenced there.
Can we scroll down this page, please, and actually over to the next page. There’s some analysis there, analysis of PEAKs, and it said:
“They are all related to different incidents of the same fundamental error message from Riposte.”
Then, “How we dealt with the problem”, it says:
“When first spotted in 2000, an avoidance action was identified and this was identified in the KEL. The advice was for SMC to monitor the associated events and then alert the branch. It isn’t clear how effective this was.”
Then it says:
“Analysis of PEAKs quoted. Which of them truly refer to same issue.
“They all relate to the same Riposte error. It isn’t clear why this re-occurred in 2010 after the Riposte fix in 2006.”
Then there’s a section on “Scope”. It says:
“The root cause of all these was a bug in Riposte that had the effect of preventing a counter from writing messages – either those being replicated to it or those generated on that counter.
“This was not always immediately obvious to the user of the counter. This could result in them thinking that some transactions which had been entered, were missing, and so they attempted to re-enter the transactions on another counter. When the offending counter was re-started, both versions of the transaction became visible and this could cause errors in the accounts.
“Attempting to balance the branch when a counter was in this state could also result in errors.”
Is that something that you remember? I’m not talking about this particular document but that kind of a summary of the problem?
Mark Jarosz: No, I’ve never seen this before. It’s the first time I have read it.
Mr Blake: But the issue, does that accurately describe for you the Riposte lock problem, or the consequences of the Riposte lock problem?
(Pause)
Mark Jarosz: So the first paragraph there about the root cause –
Mr Blake: Yes.
Mark Jarosz: – the analysis conducted, then I can see how that could be a consequence of the Riposte lock problem and, given that someone has done that analysis, it makes sense to me, yes.
Mr Blake: The reason I’m taking you to this document now is that it addresses some of the things that you said this morning, and I just want to turn over the page, please. There is some analysis of those PinICLs from 2000 and it is that first substantive paragraph, and I’m going to read it for the record. It says:
“However, on re-reading PEAK PC0126376, I can see it refers to 2 KELs (which I presumably didn’t look at back in 2010), which were raised much earlier. This shows that the Riposte issue had been initially identified back in 2000. This is made clear in KEL JBallantyne5245K and the associated PEAK PC0056922. This shows that there is a problem in Riposte such that if it loses a Thread which holds a critical lock, then Riposte grinds to a halt and the counter becomes [unstable]. The avoidance action is to restart the [computer].”
Just pausing there, do you remember advice being given to avoid it by restarting the counter? That’s something we addressed this morning, I just wanted to know if that jogged your memory at all?
Mark Jarosz: I remember you mentioning about that being stated, but it’s – it’s not advice I would ever give or agree with.
Mr Blake: But it was mentioned in the PinICL that you received?
Mark Jarosz: Yes.
Mr Blake: “The symptoms of the problem are a large number of events. The PEAK advises that if the issue occurs more than once per month, then we would need to try and reproduce the issue. The KEL also refers to PC0083101.
“Past experience shows that Escher wouldn’t consider bugs if they are not reproducible.”
Now, that’s something I asked you about this morning. Do you think that that statement is right or wrong?
Mark Jarosz: So my take on that statement is that, if the bug isn’t reproducible, then it makes progressing the root cause analysis much, much more difficult. But I’m aware that – or, on at least one occasion, when there was a bug, potential bug in the message server, Andrew Sutherland came to Bracknell to investigate it. So there’s an example, I think, where we couldn’t send him a reproducible case but he attended the facility in Bracknell to investigate.
Mr Blake: Do you think that it was common knowledge amongst those who worked on these issues that it wouldn’t really be worth troubling Escher, and perhaps not troubling you, if it was a case of a bug that wasn’t reproducible?
Mark Jarosz: Well, I think, where – I mean, the objective is to understand the issue and to close it and, in the case where that can be done, based on existing evidence, then that could be relatively straightforward. However, in many cases, a lot of effort needs to be expended in reproducing the problem to investigate it further and I can think of a number of occasions when we had to do that, so I don’t think – if a problem warrants investigation, then it needs to be investigated, and just because it’s difficult to investigate it, isn’t a reason not to investigate it.
Mr Blake: Might it sometimes be called a once-off error if it couldn’t be reproduced?
Mark Jarosz: Well, if it only ever happens once, and it can’t be reproduced then, yes, it could be labelled as it only happened once, yes.
Mr Blake: Very briefly, it says:
“The PEAK was then closed and the KEL JBallantyne5245K produced. In particular the KEL advises SMC (who monitor events from counters), that if such events are seen to phone the branch and advise them to restart the affected counter, and if they are balancing to abandon the balance until the reboot has happened as this prevents replication working correctly.”
We don’t need to spend any more time on this particular document. We can ask those who are familiar with this document about the document itself.
I want to move on to 2001 and can we look at FUJ00083592, please. So we’re now in 2001 and can we go over the page. This is an email from Brian Orzel who you mentioned earlier. It’s to a limited number of people: David Richardson, Chris Wannell, yourself, Gareth Jenkins, Lionel Higman; who are those people?
Mark Jarosz: I recognise the names, but I can’t remember their roles.
Mr Blake: Is there any significance after Gareth Jenkins’ name is says “GL” or “GI”, could those be initials, perhaps?
Mark Jarosz: I think they’re initials in the email address.
Mr Blake: This email says:
“Gents,
“It will take a little time for the new ‘users’ to bed in.”
Do you know who he is talking about there?
Mark Jarosz: No.
Mr Blake: “I am not actively working on anything in the ‘[Control-inbox]’ or ‘Parked’. If you have a pet PinICL therein that you think I should be chasing then come over and beat me up.”
He lists below a large number of PinICLs and I think there’s one – well, can you help me? If we scroll down we can see that there are some that are parked, they have various names on. Why would you be sent this?
Mark Jarosz: I think there’s a – because – the only reason I think I would be sent this is if there are some PinICLs that are assigned to me.
Mr Blake: Yes. Let’s go to page 1., it may assist us. If we look at the bottom there, it’s an email from Gareth Jenkins. Again, Gareth Jenkins directly to you:
“Mark,
“Please can you have a look through the 7 PinICLs in the list assigned to you. I suspect that many of them can either be closed or ‘Parked’. I can supply you with more details about them if you have problems in getting through to PinICL.”
What was Gareth Jenkins’ role here?
Mark Jarosz: I think he is just pointing out that some of the PinICLs are assigned to me and that they have – I assume that they have been open for a while and need to be concluded.
Mr Blake: You are one of the original recipients of the email that he is replying to on, or forwarding to you. You would have seen the original email. Why would Gareth Jenkins particularly be asking you there about seven PinICLs in the list assigned to you? What was his role in relation to your role there?
Mark Jarosz: I can’t think why he would be asking me to do this because he – no, I can’t think of a reason.
Mr Blake: If we look over the page and look at that list, there are quite a lot that say “At-Escher”. Now, would it be right to say that they couldn’t be addressed by Fujitsu because they were reliant on Escher to provide the solution in some or all of those cases?
Mark Jarosz: So the – I guess the important thing is that quite a bit of the Code used in our solution did come from Escher. So, in those cases, they would have to – they were quite rightly – if there’s a problem with the code, they would need to resolve it.
Mr Blake: Were you aware of issues obtaining code from Escher? We have heard about difficulties in obtaining the original code because of intellectual property reasons or –
Mark Jarosz: Yes, I wasn’t referring to source code. I was referring to applications. So, for example, what Escher provided us was the message server, the – at one time, there was a counter application they provided and they also provided the – the overarching application that ran on the counter, known as the desktop. So, if we identified in our testing, problems in those areas, then the right place for it to be investigated would be with Escher.
Mr Blake: Now, we have quite a few “At-Escher” and we also have some that are duplicates, I think, and also some that say “Parked”; is that right?
(Pause)
Mr Blake: So we’re there onto some “Duplicates” –
Mark Jarosz: Yes.
Mr Blake: – and then, if we keep on scrolling, I think there are quite a few that are parked as well. Yes?
Mark Jarosz: Oh, sorry, yes, I have seen both “Parked” and “Duplicates”, yes.
Mr Blake: Might some of those ones that were parked have been parked because they couldn’t be reliably reproduced at that time?
Mark Jarosz: I’m not sure of the criteria for going into a parked status, as opposed to open. I didn’t use PinICL as part of my kind of daily workflow. So I don’t know what the kind of workflow rules were for it.
Mr Blake: In relation to Gareth Jenkins – so if we go to the first page – is a fair description of this email that’s been sent to him an email that contains a list of outstanding bugs, errors and defects with Horizon?
Mark Jarosz: So the email looks to me to be a summary of PinICLs which are, I guess, in an open state, ie they haven’t been closed, and the – in terms of what they’re referring to, there could be a combination of bugs or, you know, seeking information. It’s hard just looking at the title to categorise what they fall into.
Mr Blake: Perhaps a significant list of incidents being sent to Gareth Jenkins in 2001, would you agree with that?
Mark Jarosz: Well, given that the purpose of the system was to – well, so there’s one example – it’s quite fortunate in this email Chris Wannell is pointing out that there’s a PinICL which also refers to an item which is on the RER, which is the Riposte Enhancement Register, so Chris is saying, quite rightly, it shouldn’t be a PinICL because it’s an enhancement request, as opposed to a design – as opposed to the Escher code not working as it should. So there’s just one example there, I think, of where the PinICL system is being used for something that is probably not really an incident, but I think, in general, yes, the majority would be incidents.
Mr Blake: Thank you. Can we go to FUJ00083600. Moving now to 11 May 2001. Now, this is an email, again, from Gareth Jenkins to yourself, and he says:
“I have received this PinICL.
“I know I’ve raised with you before the question of Error 82, though in the past it’s been on counters. I’m also aware that the error itself is benign, though it could result in other errors to agents (for example).”
It gives some detail there. Again, it refers in that detail to “Timeout occurred waiting for lock”, so is this, again, a Riposte lock issue?
Mark Jarosz: Yes, it is.
Mr Blake: Then if we look at the bottom, final paragraph of this page, Gareth Jenkins says there:
“What I’m really asking is for confirmation that the associated errors are indeed benign, in which case I can ensure that KELs are raised so as to suppress the reporting of them in future. It worries me that messages are failing to be inserted, however if they are being replicated, then I guess it doesn’t matter!”
Do you remember this email at all?
Mark Jarosz: I didn’t remember it until I saw the material earlier on in the week.
Mr Blake: Gareth Jenkins there is talking about a large number of errors in this particular case and he is worried that they may not be benign. Is that a fair characterisation of that final paragraph?
(Pause)
Mark Jarosz: Well, looking at the error messages he – for example, part way down the page, the third occurrence was somewhat different, the Riposte error where there’s a “RiposteStartTransaction” exception, that’s an error that hasn’t – I’m not aware we have asked Escher about that before, so it would need to be followed up with them, because it’s reporting a problem with a Riposte function.
Mr Blake: But looking, I mean, for example, at those first ones, it is very clear that some of them relate to the Riposte lock problem “Timeout occurred waiting for lock”?
Mark Jarosz: Yes.
Mr Blake: The same error that we have heard a number of times this morning. You knew Gareth Jenkins. Was his concern there genuine? Did you feel it was genuine? Did you feel his general approach to these kinds of issues was one of being worried, for example?
Mark Jarosz: So I think his concern is genuine and where he is asking for confirmation that the associated errors are indeed benign, I think it would be quite difficult to provide that confirmation, based on what I’m seeing in front of me.
Mr Blake: He is looking to you for help there, isn’t he?
Mark Jarosz: Well, he is asking me to – yes, he is, and I would have to ask Escher. I cannot recall asking Escher about that particular message, but I would have to ask them and then provide – but, in the previous explanation, I did state to Gareth that where Escher confirmed that, from a message store perspective, it’s unlikely there was an adverse impact, the – from an application point of view, it’s very important to confirm that the application is checking all the return codes.
Mr Blake: So he was aware of the information you had passed to him earlier?
Mark Jarosz: Yes.
Mr Blake: But yet he is still asking, in 2001 – I think that’s May 2001 – “Can you just really please check whether they are benign?”
Mark Jarosz: I mean, the thing is, what I can see happening, just under “The 3rd occurrence was somewhat different” section, it states – that error message states that that particular function failed, therefore – an application was trying to do something and it failed, so the – it really depends on what the consequences of that are.
So, based on what I see in front of me, I could never confirm that is benign. I would need to ask someone to look into what was happening at the time. That would be the recommendation.
Mr Blake: I think you said that you don’t recall following that up?
Mark Jarosz: No, not this one. I mean – I just cannot recall discussing this issue.
Mr Blake: Let’s move to 7 August 2001, FUJ00083608, please. So here we are, August 2001, we have an email to yourself from Gareth Jenkins. I think you are the recipient, there are a couple of people copied in there. He sends you an Escher-Dev PinICL stack, those are listed there, and can we look down at the bottom. Many of them seem to relate to Riposte. He says:
“I know the last one is assigned to me, but I sent you an email about it in July and am about to reassign it to you.
“The current situation on most of them I believe is that they are ‘one-off’ problems, and perhaps we should consider closing them. If you want help in accessing the PinICLs or their history, then please let me know.”
Again, I mean, he seems to be asking you for guidance there, isn’t he, or assistance at least?
Mark Jarosz: Yes, he is, because, in general, with the Riposte message server, at that time, we did need to liaise directly with Escher to get advice, so that’s what I would be doing.
Mr Blake: It says there:
“… I believe that they are ‘one-off’ problems …”
Does this go back to the reproducible issue that perhaps they were ones that couldn’t be reproduced?
Mark Jarosz: So I think the use of the term “one-off” applies to how often they are being observed, only once, because there could be a problem which is – which was happening regularly but it’s still difficult to reproduce it in a development environment to diagnose it further.
Mr Blake: Does that rely on somebody connecting all the dots from the one-off incidents though, to work out whether there are common themes?
Mark Jarosz: Most definitely, yes, it does. A lot of data analysis would be needed.
Mr Blake: Let’s move to 2 May, FUJ00083621. Now we’re looking at the bottom of that page, PinICL PC0075892. Again, that’s one that’s been linked to the Callendar Square issue. Let’s look over the page to page 2, and you have the customer call there, 2 May 2002. Can we scroll down a little bit. It says there:
“An unexpected error occurred while attempting to insert a message. Timeout occurred waiting for lock.”
Again, we hear that same phrase: “timeout occurred waiting for lock”.
Can we go over the page, please, towards the bottom of that page. You have John Simpkins, again, 2 May at 4.03 pm:
“These events have stopped occurring now and the Tivoli monitoring can be restarted.
“The events started at [5.29] on 1 May 2002 after the counter was rebooted. The counter produced one of these messages every 10 seconds throughout the night until ClearDesk restarted Riposte at 03.34. This cleared the lock and the system has been fine since.”
Then over the page, page 4, another substantive entry by John Simpkins:
“Appears similar to a problem we had on the correspondence servers some time back where a lock on the check point would kill agents.
“Attached application log as evidence. Passing to development for comments.”
Then we look at page 1 and this is, again, a PinICL that’s sent to Gareth Jenkins and, again, it’s got Gareth Jenkins asking you follow-up questions. This time we’re now in May 2002. Again, Gareth Jenkins seems to be asking you for your opinion. He says:
“Any thoughts on this one? Unless there is something obvious to investigate I suggest we will probably need to write this off as a ‘one-off’. Is it worth trying to find out why the machine was rebooted?”
So he doesn’t seem there to be asking you simply to make contact with Riposte. He does seem to be asking you for your substantive opinion on a particular problem, doesn’t he?
(Pause)
Mark Jarosz: In this case, I think we would need to confirm what those – the right course of action would be to seek confirmation from Escher what those error messages mean and what the consequences are.
Mr Blake: Time and time again we have seen emails from Gareth Jenkins to yourself. He is not just asking you to contact Escher and be the message man. I mean, he is really asking you for your thoughts on this particular problem.
Mark Jarosz: But the only way I could contribute to the conversation with Gareth would be to liaise with Escher because, without any documentation on their message server, the only way I can gain knowledge is by speaking with Escher.
Mr Blake: He is saying there that he will probably need to write it off as a one-off. Again, I mean, this is a problem with Riposte in the error message. I imagine subpostmasters will be asking how many one-offs makes something not a one-off.
Mark Jarosz: What isn’t in the email is any context about what the application was doing at the time, if anything.
Mr Blake: This phase is focused on rollout 2000, et cetera. We know that the Callendar Square bug continued until at least 2006. There was an S90 software fix; is that something you’re aware of?
Mark Jarosz: No.
Mr Blake: It had the potential to cause discrepancies. Shouldn’t this Riposte lock issue have been front and centre of your witness statement?
Mark Jarosz: No – so when I produced the initial witness statement at the time, my recollection of the Riposte errors were as I described: requesting information from Escher as to what they mean and what the consequences could be.
Mr Blake: The picture that’s built up this morning is that you were quite involved in this particular issue, weren’t you?
Mark Jarosz: Even though we have focused on this, it was a very small part of my normal role within the programme.
Mr Blake: These continued problems with the Riposte lock, do you know if anyone was feeding those problems back to the Post Office?
Mark Jarosz: I don’t know and I don’t think I would know.
Mr Blake: Did you ever –
Sir Wyn Williams: Could the document be taken down, please?
Mr Blake: Did you ever speak to any subpostmasters directly about issues with Riposte?
Mark Jarosz: No.
Mr Blake: Thank you, sir. Those are all my questions. Mr Jacobs, I think, is first.
Sir Wyn Williams: Over to you, Mr Jacobs.
Questioned by Mr Jacobs
Mr Jacobs: Thank you, sir. Can I just check that you can see me and that you can hear me?
Sir Wyn Williams: I can, yes.
Mr Jacobs: Thank you.
Mr Jarosz, I have some questions for you on behalf of 153 subpostmasters who were pursued by Post Office for shortfalls that were apparent which they couldn’t check.
I want to ask you about replication. In your statement at paragraph 21(d), if we could call that up on the screen, it’s WITN04810100, page 10 of 22. Thank you. I’m just waiting for it to come up on the screen. Thank you.
So you talk about an approach taken whereby messages were replicated and:
“… the system created multiple copies of a message on each message store.”
Is that right?
Mark Jarosz: So on each counter – so on each counter apart – so on each counter there was a single message store.
Mr Jacobs: Yes.
Mark Jarosz: And if there are two or more counters in a branch then each of those counters would have its own message store and the Riposte behaviour was to – if a message got created on the third counter, it will be replicated to every other counter in the branch.
Mr Jacobs: Right, and I think the position is that, if one counter was down, the other counter would “know” the message on the counter that wasn’t functioning.
Mark Jarosz: So in that scenario, if replication is working correctly, then each counter gets a copy of messages from every other counter and also from the correspondence servers in the data centre, so within a given message store, yes, you see messages for every counter and the correspondence service.
Mr Jacobs: The reason I have been asked to ask this question is because many of our clients, when they gave evidence in the Inquiry in February to March of this year, came up with quite a similar issue where they would have a shortfall, say for example £2,000, they would go into the system to try and resolve it and it would come up at £4,000, then it would come up as £8,000, and it would keep replicating.
The question I have is: could it be the case that these replicated shortfalls arose from the replication system that you have described not working correctly in addition to or alternatively to bugs, errors and defects that we know about?
Mark Jarosz: So I think I would answer in two parts. The first part is, if the replication wasn’t working correctly, then there could be a number of scenarios. For example, some counters would be missing messages from other counters, possibly because of a – the network in the branch was partitioned. So I think a plausible scenario, which I can envisage would be in a multi-counter office, if a network gets partitioned anyway, then some counters won’t be able to replicate to other counters.
Now, in terms of how that would manifest itself, it would mean that the counters which cannot reach a gateway have no online communication with the data centre. So there might be some observable incident as a result of that. It depends what proportion of transactions were online and what proportion were performed locally.
Mr Jacobs: If that did happen, if the system got stuck in this way and there was no connectivity, I think your evidence is that there was something called a gateway node, so that everything would sort of feedback in once it was restored. Is there a possibility, is it plausible, that that part of the process could lead to subpostmasters having their shortfalls doubling up through a malfunction of this part of the system?
Mark Jarosz: So the special role of the gateway is it is the only counter which communicates with the correspondence servers at the data centre. So in the scenario I described of the network being partitioned, what that would mean is that the gateway and some other counters would have a – would have messages being created and communicating with the data centre, whereas some other counters would be isolated and, therefore, then messages wouldn’t be replicated until the network was restored, so there would be different messages in different parts of the network.
In terms of the consequences of that on the application, unfortunately I can’t – I have no expertise in that, how the application would interpret that scenario. But, certainly, from a network point of view that could happen and the thing I would mention, of course, is in a single counter office, there’s only one counter, it’s the gateway counter and, in that case, there’s two Riposte message servers on the counter replicating to each other. And the reason for that is, should that counter fail, then it has a removable drive so the replacement one can be initialised from that.
Mr Jacobs: So I think what you’re saying, and correct me if I am wrong, is that, although you’re not able to be absolutely clear, it’s possible that the scenario that I have described could have arisen from a malfunction of this part of the system?
Mark Jarosz: Yes, definitely, because, even though in my witness statement I state how it’s designed to work, clearly networks do fail for periods of time and therefore this partitioning can occur.
Mr Jacobs: Thank you. The next question that I have for you relates to connectivity in remote areas and this is in relation to paragraph 38(b) of your statement, which is, again the same reference, WITN04810100, paragraph 38(b) please, page 17 of 22. We can see towards the bottom of that section you say:
“I recall there were about 140 branches where we could not use ISDN as the branches were very remote. In those cases, as ISDN was not available, we used VSAT …”
We know from above that means “very small aperture terminal”:
“… as an alternate means of connection. VSAT is, effectively, a satellite connection and, as with any network solution, its reliability depends on the context in which it is deployed. For instance, VSAT reliability can be affected by inclement weather.”
Again, the reason I’m asking this question is because it arises from the experiences of some of our clients, who say that they experienced power outages and shortfalls arose often after there were power outages.
Now, what I wanted to ask you is: you said here that VSAT reliability can be affected by inclement weather. What sort of weather conditions would affect that reliability?
Mark Jarosz: Rain and snow, for example, because they attenuate the signal.
Mr Jacobs: So this is to do with …
You say that:
“… as with any network [position] its reliability depends on the context in which it is deployed.”
What were the other issues that affected VSAT reliability?
Mark Jarosz: So as well as the weather conditions, the VSAT service that we used was from a single provider.
Mr Jacobs: Yes.
Mark Jarosz: Slightly different to the ISDN service, where, because it’s geographically distributed, there are multiple exchanges being used. So if this provider, for example, has some problem in their network, then it could affect all or multiple branches that relied on VSAT for communications, for the period of time that that problem persisted.
Mr Jacobs: Okay. Do you accept because of this, those who were in rural areas were more vulnerable to difficulties with the system than other subpostmasters?
Mark Jarosz: The – so I’m trying to think what characteristics would be affected by rural areas, so certainly the … I’m trying to think of a characteristic of the network which was affected by distance from exchange or VSAT.
(Pause)
Mark Jarosz: I’m struggling to come up with a plausible scenario which would differentiate the network characteristics. There may be one, I just cannot think of one off the top of my head.
Mr Jacobs: Well, I will move to my next question. Could an unstable connection affect post office systems or balances?
Mark Jarosz: Well, so an unstable connection would – we’re talking about the connection from the gateway now, into the data centre –
Mr Jacobs: Yes?
Mark Jarosz: – as opposed to within the branch? So it would certainly affect message replication between the branch and the data centre and the – though it would manifest itself as where either the data centre or the branch need to communicate with each other because they need to exchange messages for some application reason, but they are unable to, or it happened intermittently, so that would certainly happen and, again, the consequences of that on the application obviously depend on the application, but yes.
Mr Jacobs: Thank you. I’m just going to see if I have anything else to ask.
That covers all the questions I have. Thank you very much.
Mark Jarosz: Thank you.
Sir Wyn Williams: Do we have any other questions?
Ms Page: Yes, sir, some questions from me please, sir.
Sir Wyn Williams: Very well.
Questioned by Ms Page
Ms Page: I’m Flora Page. I’m also acting for a number of the subpostmaster Core Participants and I’m also going to focus on what I understand to have been your responsibility, which was the network solution, and your – that means, doesn’t it, that you were responsible for the design of the counters communicating with the central data hubs; is that right?
Mark Jarosz: Yes, for the network service that we provided to enable that communication to take place.
Ms Page: Have you had a chance to look at a section of the report from Mr Charles Cipione, which he headed with the title “Many Post Office branches were disconnected from the central system during national rollout”? Does that ring a bell at all? We can bring it up.
Mark Jarosz: No, it doesn’t, but – I mean, what I would say is in general that branches being disconnected from the central system would happen when – for example, if it was an ISDN outage, which is why we had other solutions in place to deal with that.
Ms Page: Well, let’s just, if we can, we will bring up EXPG0000001, and this is Mr Cipione’s report. If we look at page 83, please.
So we see that heading there, it takes perhaps a little bit of unpacking but he talks about how the design used – in that second paragraph he talks about the design feature was a telecommunications system which depended on ISDN or, in some cases, satellite links and I think that ties up with what you have already told us, doesn’t it?
Mark Jarosz: Yes.
Ms Page: It says at 10.1.3 that:
“The Monthly Reports indicate throughout 1998 and 1999 that ICL Pathway was concerned with their ability to effectuate this design feature: they were concerned with BT’s coverage of the UK as well as other technical issues related to their standards.”
Then it says, in the following paragraph, 10.1.4:
“During the national rollout these problems were realised. Hardware, network availability and user issues combined to create a situation where ICL Pathway was occupied with a higher than expected amount of non-polling branches.”
He explains there are two problems associated with that:
“This was problematic because [it] relied on the telecommunication design aspect … to collate and centralise information on all the activity of the branches, but also to allow for efficient updates of software to the branches.”
Does that make sense to you?
Mark Jarosz: It does, yes.
Ms Page: All right, so the “polling”, that’s just a terminology for the branches connecting to the central servers, isn’t it?
Mark Jarosz: Yes.
Ms Page: He then goes on a bit further on in this section to provide statistics on the numbers of branches which were not polling or didn’t poll for significant periods of time. He has already identified there, hasn’t he, the issues that result from that: the former one being the data not actually managing up, so things not getting to the central data, which should have done, from the counters; is that fair?
Mark Jarosz: Yes, if they were disconnected then that would happen, yes.
Ms Page: Were you conscious at the time of rollout, and surely you should have been, that non-polling was an issue?
Mark Jarosz: I wasn’t conscious that there was a higher – I wasn’t conscious that it was a higher than expected amount of non-polling branches but non-polling was a consequence of the network solution because there was no resilient networking – at this point in time, so I’m thinking in this period of time up to 2000, there was no network resilience for branches, so if the primary network service wasn’t functioning, then there would be non-polling.
This was one of the reasons for introducing the manual back up process.
Ms Page: When was that introduced?
Mark Jarosz: I’m not sure when that was deployed, but this was the process when an engineer would go to the branch and use alternative telecommunications services, either wireless or PSTN, to connect the branch to the data centre.
Ms Page: You can’t tell us when that was? Are we talking months, years after rollout?
Mark Jarosz: I would have to check when it was deployed, but it – is this national – I’m struggling to understand –
Ms Page: National rollout was sort of through 1999 and 2000. 2000 was when it really began in a big way.
Mark Jarosz: Okay. So I would have to check when this manual solution I explained was deployed. I just don’t know when it was deployed, but it –
Ms Page: No, all right. Well, is it possible that non-polling would have continued as an issue until Horizon Online or is that wrong?
Mark Jarosz: So the original reason for using ISDN as a network technology, one of the justifications was that most of the transactions didn’t require an online connection to be carried out, albeit they did need to synchronise.
When the change was made to not do the benefit transactions but to move to Network Banking, then the whole network approach changed and, at that point, we were looking at having backup technology integrated, so there will be a primary network type and a backup network in each branch.
Ms Page: So we’re talking, are we, about the Post Office’s attempts to move into different areas because the Benefits Agency revenue stream was no longer –
Mark Jarosz: Yes.
Ms Page: – going to be there?
Mark Jarosz: And the consequence on the network being that the network had to be there for those transactions to take place, as opposed to it was more a batch system where the transactions could take place and then get synchronised later. So, yes.
Ms Page: So Network Banking was going to require being constantly on, was it, as opposed to the intermittent design?
Mark Jarosz: Yes.
Ms Page: Did that ever come to pass before Horizon Online?
Mark Jarosz: Yes, it did.
Ms Page: Can you give us an idea of when that was?
Mark Jarosz: So I’ve got the timescales here, I can just look them up.
(Pause)
Mark Jarosz: So the network changes which introduced – the diagram I’m looking at here starts in 2006, so I’m just – I don’t have the information about exactly what happened before that but, certainly, in 2006 is when we started rolling out the branch network device which had integrated backup.
Ms Page: So that was going to be fully on all the time, instead of the polling issue?
Mark Jarosz: Yes, most definitely, and, in fact, we did introduce fully on much earlier than that. As soon as we went to online banking, we moved away from ISDN intermittent to ISDN nailed up.
Ms Page: Again, can you say when that was?
Mark Jarosz: Not accurately, not without checking, but it would have been prior to introduction of any online banking because it wouldn’t have been possible to do it over the ISDN network on demand.
Ms Page: All right. So for a period of, presumably, some years, at least, after rollout in 2000, there was still this intermittent service with the occasional non-polling incidents; is that right?
Mark Jarosz: Yes.
Ms Page: All right. Well, let me just then – just a few questions to bottom out what non-polling meant and how it would have affected subpostmasters.
So if we look at page 87 of the document that’s up and we scroll down, thank you, to a summary of – that one that’s actually at the bottom of the page, so we can stay there.
This is a list of extracts from PinICLs and the one that’s dated 4 January 2000 explains a sort of a typical example:
“‘This office is still not polling and hasn’t polled for 11 days – please resolve ASAP.’ ‘Missing objects relating to EPOSSRec were inserted today by P Carroll. The PO should disappear from the non-polling report tomorrow.’”
So what we’re seeing there is the effect of non-polling is that one can have missing objects, in other words missing transactions; is that right?
Mark Jarosz: The …
(Pause)
Mark Jarosz: Based on the non-polling report, showing that this particular post office wasn’t able to communicate with the data centre, then any objects created in the data centre would not have made it to the post office and similarly in the other direction, so that’s – that’s what would happen if there was no communication.
Ms Page: So the result here is that objects have had to be inserted, in other words transactions have had to be put into the accounts, haven’t they?
Mark Jarosz: Well, I think – so my immediate thought on reading this is that I recall that, after a number of days of non-polling, there was meant to be a process in place to try and synchronise the post office with the data centre, so that’s what would have I expected to be at the normal as-designed solution behaviour for this.
In terms of what’s happened here, clearly that didn’t take place, or wasn’t successful, so I can see that the individual – I know who he is – is stating that he had to put in some missing data. What I cannot tell from this is whether that missing data was something he had to insert in the data centre, but I – on the basis that the branch is non-polling, it would mean that it would have to be there because he can’t communicate with the branch.
But what I cannot tell from reading this is whether this is an approved workaround or whether this is a one-off because the as-designed solution would be for someone to attend that office with the special laptop to attempt for it to synchronise with the data centre.
Ms Page: Was there a process for making sure this person with the special laptop arrived?
Mark Jarosz: Yes. There was a whole solution around this. I think it was called Day D solution.
Ms Page: Say that again?
Mark Jarosz: I think – I recall it was called the Day D solution.
Ms Page: Day D solution?
Mark Jarosz: Yes.
Ms Page: Would the subpostmaster in the run-up to this have received any alert or any message?
Mark Jarosz: I’m not sure what the operational service was around how this was deployed. I mean, clearly, to gain access to the post office, there would have to be some kind of communication, but I’m not sure what the service process was.
Ms Page: Absent a human intervention, somebody arriving with the special laptop, was there any system built in, automated, if you like, that would tell postmasters when they weren’t polling?
Mark Jarosz: I don’t know. There certainly could have been, very easily, but I don’t know if that was actually deployed on the counter because, clearly, the – any – it would be very easy on the counter to detect that this is happening, but whether it was put in place or not, I don’t know.
Ms Page: Who would have been responsible for that?
Mark Jarosz: So that would be as part of the counter development team. So the – so I think that would be – at the time, Gareth was the counter and Riposte TDA so he would have been aware of that, or it could have been one of the application people. I’m really not sure even if there was anything put in place like that.
Ms Page: Was there any liaison with your team over thinking through the implications of this, so that – your team obviously being responsible for the network side of it and Gareth’s team thinking about it from the point of view of the counter application, was there effective liaison to make sure that subpostmasters would receive the right sort of messages that might say, for example, “You haven’t polled for a number of days, there’s a risk of missing transactions”?
Mark Jarosz: I’m not sure if that took place or not.
Ms Page: You don’t recall, anyway, having those kind of conversations?
Mark Jarosz: No, and I probably wouldn’t have been involved in them if there were, so I wouldn’t expect to be involved in them.
Ms Page: Who would have been involved with them on the network side?
Mark Jarosz: So we had network designers. At the time of doing that solution, that was David Tanner, so from a network design point of view, it would have been him. It would also have been customer service because, this is a service-related matter, so they would have been involved.
Ms Page: Thank you.
Mr Blake: Thank you, sir. I don’t think there are any other questions.
Sir Wyn Williams: All right.
Well, thank you, Mr Jarosz, for coming to the Inquiry and answering all the questions which were put to you. I’m grateful.
Mark Jarosz: Thank you.
Mr Blake: Thank you, sir. Are you content for us to all take an hour’s lunch now rather than starting the next witness and interrupting?
Sir Wyn Williams: Of course, yes.
Mr Blake: Thank you, sir. Perhaps we could come back at 1.30.
Sir Wyn Williams: Yes, by all means.
Mr Blake: Thank you very much.
(12.32 pm)
(The luncheon adjournment)
(1.30 pm)
Ms Hodge: Good afternoon, sir.
Sir Wyn Williams: Good afternoon.
Ms Hodge: We’re just waiting for the next witness to attend.
Sir Wyn Williams: Yes, that’s fine.
(Pause)
Ms Hodge: Sir, our next witness is Mr Jeram. Please could the witness be sworn.
Mr Peter Jeram
MR PETER JERAM (sworn).
Questioned by Ms Hodge
Ms Hodge: Please give your full name.
Mr Peter Jeram: Sorry?
Ms Hodge: Please give your full name?
Mr Peter Jeram: Peter Ernest Jeram.
Ms Hodge: Mr Jeram, you should have in front of you a witness statement dated 6 August of this year –
Mr Peter Jeram: Yes.
Ms Hodge: – is that right? The statement runs to nine pages. Could I ask you please to turn to page 8 of that statement.
Mr Peter Jeram: Yes.
Ms Hodge: Do you see your signature there –
Mr Peter Jeram: I do, yes.
Ms Hodge: – at the bottom of the statement. Is the content of the statement true to the best of your knowledge and belief?
Mr Peter Jeram: Yes. I’ve got one comment on it, if that’s okay? In section 15, when I read that again, when it says in there about the cash account, and I made a statement saying “and cannot therefore comment on whether there were issues”, I was talking about issues that we didn’t know about in my role and support in the end to end and MOT. I did know about issues that were found and then resolved. I just wasn’t sure that was clear on that statement.
Ms Hodge: So what you’re saying your evidence is, that at paragraph 15 of your statement, you were saying that you were not aware of issues of which you were not aware; is that, in effect, your correction?
Mr Peter Jeram: I guess so, yes. This implied I didn’t know about anything and we did have issues and we did correct issues. So … yes.
Ms Hodge: Thank you. I’m going to begin by asking you some brief questions about your recruitment by ICL Pathway. You joined ICL Pathway as a release project manager in approximately 1997; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: At that stage, you were not an employee of ICL Pathway –
Mr Peter Jeram: Correct.
Ms Hodge: – but you had been recruited to join the programme via a IT consultancy; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: You later became a permanent employee of what became known as Fujitsu Services Limited in or around April 2003; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: You remain employed by Fujitsu today; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: But not on projects related to Horizon, I understand?
Mr Peter Jeram: No.
Ms Hodge: It is in your capacity as a current employee of Fujitsu, who had direct involvement in the matters to which this Inquiry relates, that you were invited to provide a witness statement to the Inquiry on behalf of Fujitsu; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: The purpose of that statement was to assist the Inquiry with the matters canvassed in two Rule 9 requests, the first dated 11 March of this year; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: And the second, 1 July?
Mr Peter Jeram: Okay.
Ms Hodge: Those requests covered a range of issues, which included issues identified in the development of the cash account function Horizon, and you have referred just now to paragraph 15 of your statement –
Mr Peter Jeram: Yes.
Ms Hodge: – which was directed at that, as well as the accuracy and integrity of the data recorded and processed on the Horizon System, and the extent to which deficiencies with Horizon were capable of causing apparent discrepancies or shortfalls in branch accounts.
Those were the three areas canvassed in those requests, were they not?
Mr Peter Jeram: Sorry, I don’t remember the detail of the request. I was certainly asked some questions which I answered.
Ms Hodge: Bearing in mind what you have just told us about paragraph 15 of your statement, do you consider you have been candid in your statement to the Inquiry about your knowledge of technical issues with Horizon at the time?
Mr Peter Jeram: I would say I remember more now from the bundles I have been provided than maybe I did at the time of the statement.
Ms Hodge: Before you finalised your statement, you were invited to refresh your memory from the contemporaneous records held by Fujitsu, were you not?
Mr Peter Jeram: I was certainly given some documents to remind myself on things, yes.
Ms Hodge: You had access to all of the documents in Fujitsu’s possession, did you not, that were relevant to your involvement?
Mr Peter Jeram: I don’t know. I was certainly – had access to some documents that were provided to me, yes.
Ms Hodge: Did you ask to be provided with all documents that were relevant to your involvement in the period prior to the rollout of Horizon?
Mr Peter Jeram: When I had some questions on things I asked and was provided with a document, yes.
Ms Hodge: I wonder if you could help us then. How is it that you came not to mention the issues that you say you have now come to understand in the recent disclosure that’s been provided to you?
Mr Peter Jeram: It’s more a case of reading the wording that I put in there because, for example, I got involved in the end to end testing and the model office rehearsals and testing with Post Office and, through that, there would have been incidents that were raised on the cash account and incidents that were cleared, so I would have had the visibility of those taking place at that time.
Ms Hodge: You specifically mention in your statement, at paragraph 26, that you were aware of a number of formal internal audits of the Horizon System; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: Did you ask to see copies of those audit reports before finalising your statement?
Mr Peter Jeram: Yes, I did see some.
Ms Hodge: We will return to some of those a little later. In your role as release project manager, I understand you were responsible for project managing the release of software by ICL Pathway into the live estate; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: You have explained in your statement that this was not a technical role, as far as you were concerned –
Mr Peter Jeram: Yes.
Ms Hodge: – and that you relied on those who did have the relevant technical expertise to bring technical matters to your attention; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: You have also stated you were not directly involved on the technical side of the development of the project; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: You were, however, notified of significant technical developments and issues which affected the timing and release of software; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: Presumably, knowledge of such technical issues would have been critical for you to perform your role as a project manager?
Mr Peter Jeram: To a certain level, yes.
Ms Hodge: Though not a technical expert, you presumably had quite a high level understanding of the purpose and function of the key components of Horizon; is that correct?
Mr Peter Jeram: Probably at that time, yes.
Ms Hodge: You would have known, therefore – but please correct me if I’m assuming too much – but I presume you would have known that the Electronic Point of Sale Service, one of the key components of the Horizon System, was responsible for recording and processing all of the transactions carried out within the Post Office branch by customers purchasing goods and products of the Post Office; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: You would have known it was responsible for balancing receipts and payments –
Mr Peter Jeram: Yes.
Ms Hodge: – and for producing what was known as the cash account; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: Presumably, you knew that the essential function of the cash account was to produce the definitive weekly summary of all the transactions recorded within the post office branch –
Mr Peter Jeram: Yes.
Ms Hodge: – and that the cash account function, therefore, served an essential accounting function, both for the Post Office and for its agents who were using the system?
Mr Peter Jeram: Yes.
Ms Hodge: I would like to ask you some questions about a report which was produced in September 1998 on completion of what was known as the EPOSS PinICL task force. The report to which I’m referring bears the unique reference number FUJ00080690. You have had an opportunity to read this report, have you?
Mr Peter Jeram: Yes.
Ms Hodge: Do you recall being shown a copy of this report at or around the time it was produced in September 1998?
Mr Peter Jeram: No.
Ms Hodge: Were you aware in the summer of 1998 that the volume of PinICLs recorded against the Horizon product was very high?
Mr Peter Jeram: The timing not sure but, yes, I know there were lots of PinICLs at some stage.
Ms Hodge: Did you know at the time that the PinICL count was sufficiently high that the task force had been established in an effort to reduce it?
Mr Peter Jeram: Yes, I think I do.
Ms Hodge: Were you made aware, on completion of the task force, about the concerns which had been expressed by those with the relevant technical expertise about the quality of the EPOSS code?
Mr Peter Jeram: There was certainly a discussion about – yes, the quality of the product.
Ms Hodge: Was that a discussion that you had in one of the development directors meetings, which you attended with Terry Austin?
Mr Peter Jeram: Probably.
Ms Hodge: What exactly did you understand to be the nature of the concerns about the quality of the EPOSS code?
Mr Peter Jeram: With the large number of PinICLs that had been raised through the testing services.
Ms Hodge: That indicated what, as you understood it?
Mr Peter Jeram: That the product was of questionable quality.
Ms Hodge: Were you aware that those with relevant technical expertise had expressed fears that the application of PinICL fixes was likely to lead to yet further degradation of the quality of the EPOSS code?
Mr Peter Jeram: Where – we were aware at the time that would be a risk with the number of changes that were being made.
Ms Hodge: So that was something of which you would have been aware at the time?
Mr Peter Jeram: Yes.
Ms Hodge: You mentioned, when confirming the content of your statement, that you had some oversight of model office and end to end testing; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: We know that you participated in a review in early December 1998 of the PinICLs raised during what was known as MOR3 and end to end testing, which had been carried out in November of that year; that’s right, isn’t it?
Mr Peter Jeram: Yes, to my knowledge, yes.
Ms Hodge: You have been shown a copy of the memorandum that was produced by Andrew Simpkins on 4 December 1998 in connection with that review, have you not?
Mr Peter Jeram: I have looked at all the things sent to me, so if that was one of them, yes.
Ms Hodge: The document to which I’m referring is POL00028429, please. We can see that you are named as one of a number of recipients of that memorandum, are you not?
Mr Peter Jeram: Yes.
Ms Hodge: Have you taken the opportunity to refresh your memory of the document?
Mr Peter Jeram: This particular one versus the others, I can’t remember exactly what this one said.
Ms Hodge: The memorandum canvasses a number of issues that were identified during the review. If we could scroll down, please, to the first page, a little bit lower on the first page. Under the heading “Progress this Week”, Mr Simpkins confirms:
“As you are aware Horizon, TIP [which would be transaction information processing] and Pathway have carried out a comprehensive and detailed analysis this week of PinICLs arising from MOR3 and E2E …”
End to end. So those two testing cycles; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: “… (and outstanding faults from previous phases). We would like to thank Pete Jeram for his active support to this review. I attach a copy of the summary totals and of the full PinICL analysis pack.”
If we go on a page please to the second page, under the heading “Testing Issues”, it reads:
“Since tabling the paper on the ‘Key Problem Area Analysis’ at the Checkpoint meeting on 18th November good progress has been made on most of the 9 areas identified. We will reissue this summary next week showing the current action points. Specific concerns that have been confirmed by the PinICL review include …”
Then we can see a list of issues. Firstly, those on the transaction information processing interface, these include “Inconsistencies between the transaction file totals and the cash account”, that’s the third bullet point. Can you see that?
Mr Peter Jeram: Yes. I wouldn’t have been involved in the detail of these things. The action I was doing was making sure there was a review and that everybody was being very open and sharing them.
Ms Hodge: Well, we can see you participated in that review –
Mr Peter Jeram: Yes.
Ms Hodge: – and Mr Simpkins expressed his thanks to you for assisting.
Mr Peter Jeram: But it wouldn’t have been my thanks from diagnosing the issues, or the things.
Ms Hodge: No, forgive me, I’m not suggesting that you would have had a detailed technical awareness of the underlying causes of these issues, but you would have been aware, surely, by virtue of that review and the receipt of this memo, that issues of this nature were being discussed at the time –
Mr Peter Jeram: Yes.
Ms Hodge: – in December 1998?
Mr Peter Jeram: Yes.
Ms Hodge: So we have there, at the third bullet point:
“Inconsistencies between the transaction file totals and cash account …”
At the fourth bullet point:
“‘Lost’ BES [that would be Benefit Encashment Service] transactions on the transaction file.”
A little further down, there is reference under the heading “On the Counter” to a number of incidents around stock unit balancing. Then, if we scroll down a little further, please, under the heading “Other Issues”, we see a number listed, the first of which are “cash account balances”, and there’s reference there to a constructive joint meeting on the reasons for imbalances and the action being taken to address these.
Mr Peter Jeram: Yes.
Ms Hodge: I think we’re agreed that, in December 1998, these were all issues that were certainly on your radar?
Mr Peter Jeram: Yes.
Ms Hodge: You also knew, did you not, that there remained quite significant concerns about Horizon’s accounting integrity at the point at which the system was accepted by the Post Office in late September 1999.
Mr Peter Jeram: Although I wasn’t directly involved in the acceptance, I know there were issues that were going through discussion, exactly.
Ms Hodge: The concerns about Horizon’s lack of accounting integrity were sufficiently serious at that stage, were they not, that ICL Pathway had agreed to produce a new piece of software to perform reconciliation checks? You were aware of that, were you not?
Mr Peter Jeram: Yes, that was towards the end of 2000 maybe. No – yes, by the end of 1999, yes, yes.
Ms Hodge: The purpose of that software, known as the accounting integrity control release, was to detect cash account imbalances and to produce reports to enable them to be rectified –
Mr Peter Jeram: Yes.
Ms Hodge: – is that not correct?
Mr Peter Jeram: Yes.
Ms Hodge: You were, in fact, responsible for project managing the release of that piece of software, were you not?
Mr Peter Jeram: Yes.
Ms Hodge: We can see that if we pull up document FUJ00118156 please. Forgive me, it’s 156 please.
(Pause)
Ms Hodge: Thank you. Sorry, that reference error was my fault.
This document is described as a process release note.
Mr Peter Jeram: Mm-hm.
Ms Hodge: It is dated 29 October 1999. We can see at the top it is version 0.1. It:
“Provides a definition of the CSR+ Increment 2.2 [relating] to Acceptance Incident 376 Release for [Post Office Counters Limited].”
Mr Peter Jeram: Yes.
Ms Hodge: A document that was reviewed by you –
Mr Peter Jeram: Mm-hm.
Ms Hodge: – and to which you contributed at the time?
Mr Peter Jeram: Yes.
Ms Hodge: Before we move on, please, to another topic, I would be grateful if you could assist me with one further document. This is document FUJ00118175, please. This is a document which was produced to the Inquiry by ICL Pathway. You have been shown a copy of this document, I believe.
Mr Peter Jeram: I have, yes.
Ms Hodge: Have you taken the opportunity to read it?
Mr Peter Jeram: I have. I wouldn’t say I fully understand it but …
Ms Hodge: It’s clear from its title that it relates to EPOSS reconciliation issues and Acceptance Incident number 376.
Mr Peter Jeram: Yes.
Ms Hodge: There’s an entry at the top which indicates that comments have been added to the document by “P JP” can you help us with who is that a reference to you?
Mr Peter Jeram: I don’t think – when I read it and I saw that I thought it might be John Pope.
Ms Hodge: Thank you. I would like now, if I may please, to turn to the development audit of the Core System Release Plus, which was conducted in September 1999. This was an audit carried out by Jan Holmes, Pathway audit manager, who produced a report recording his findings in late October 1999.
Mr Peter Jeram: Yes.
Ms Hodge: You were made aware at the time of the findings of that audit, were you not?
Mr Peter Jeram: Yes.
Ms Hodge: Do you recall reading the audit report?
Mr Peter Jeram: Yes.
Ms Hodge: You have recently been provided with a copy. Have you taken the opportunity to refresh your memory –
Mr Peter Jeram: Yes.
Ms Hodge: – of its contents. It bears the reference FUJ00079782, please. Have you read, in particular, the section of the report at pages 19 and 20 which addressed the author’s findings in relation to EPOSS?
Mr Peter Jeram: I read the report, so I would have gone through that as well, yes.
Ms Hodge: Whether or not you were shown a copy of the report into the EPOSS PinICL task force in September 1998, you would have known, upon reading this audit report, that the EPOSS PinICL task force report had the previous year called into question the maintainability and resilience of the EPOSS code –
Mr Peter Jeram: Yes.
Ms Hodge: – and that was by reason of the high number of PinICL fixes which had been applied to the EPOSS product –
Mr Peter Jeram: Yes.
Ms Hodge: – that’s correct, isn’t it? What’s more, you would have also known, on reading this report, that, since completion of the task force, nearly 1,000 PinICLs had been raised against the EPOSS product and that the application of fixes to address those faults was bound to have worsened the quality of the code?
Mr Peter Jeram: There was that risk, yes.
Ms Hodge: Jan Holmes’ concerns about the quality of the EPOSS product were sufficiently grave that he recommended that consideration be given to redesigning or rewriting EPOSS?
Mr Peter Jeram: Yes.
Ms Hodge: You were aware of that?
Mr Peter Jeram: Yes, I saw that in the report, yes.
Ms Hodge: This wasn’t the first occasion on which that recommendation had been made, had it?
Mr Peter Jeram: I don’t remember.
Ms Hodge: We know from reading this audit report that an earlier report addressing the quality of the EPOSS product had been produced by Pathway on 21 September 1999. Were you aware of that report?
Mr Peter Jeram: I don’t know.
Ms Hodge: Perhaps if we could turn to page 2, please. If we scroll down, please, thank you. Under the heading “0.3 Associated documents”, there’s a reference at point 7 to a report on EPOSS solutions, dated 21 September 1999. Were you aware of that report?
Mr Peter Jeram: I don’t remember that report.
Ms Hodge: We see that report referenced in this audit. If we could go to page 20, please. In the box there, the audit states that:
“The EPOSS Solutions Report [document number 7 in the associated documents we saw just a moment ago] made specific recommendations to consider the redesign and rewrite of EPOSS, in part or in whole, to address the then known shortcomings.”
So that recommendation was first made on 21 September 1999. Do you know whether or not a copy of that report was provided to Post Office Counters prior to their decision to accept the Horizon decision in late September?
Mr Peter Jeram: I don’t know.
Ms Hodge: Do you consider that a copy of that report should have been provided to Post Office Counters to inform their decision about acceptance of the Horizon System?
Mr Peter Jeram: I think so, along with the testing from it, yes.
Ms Hodge: Who do you think was responsible for ensuring that was done within ICL Pathway?
Mr Peter Jeram: It probably would have been done through one of the reviews that Mike, Terry and I were at.
Ms Hodge: Forgive me, but by “reviews” do you mean internal reviews –
Mr Peter Jeram: No, the meetings we had with Post Office.
Ms Hodge: With Post Office?
Mr Peter Jeram: Yes. I don’t think we did, but that, I guess, would be the place that it was shared.
Ms Hodge: That’s where it ought to have been shared, is your view?
Mr Peter Jeram: Yes.
Ms Hodge: When you received a copy of this CSR+ development report in late October 1999, did you take any steps to bring its findings and recommendations to the attention of Post Office Counters?
Mr Peter Jeram: I don’t remember doing so.
Ms Hodge: Do you consider that a copy of the CSR development audit report should have been provided to Post Office Counters to inform their decision about the resolution of Acceptance Incident number 376?
Mr Peter Jeram: I think we concentrated more on the testing that showed that it was working, necessarily, than the report, but as we had quite an open relationship then – yes, I don’t know why we wouldn’t have shared it.
Ms Hodge: Is it your evidence that you think you would have shared it at the time?
Mr Peter Jeram: No, my evidence is I don’t know.
Ms Hodge: You just don’t know. The CSR+ development audit report was supported by a schedule of corrective actions in which the recommendations resulting from the audit were recorded and agreed corrective actions were documented. Were you aware of that?
Mr Peter Jeram: Yes.
Ms Hodge: One of those recommendations we can see here was that, in light of the continued evidence of poor product quality”, that is to say in the EPOSS product, that the recommendations to consider the redesign and rewrite of EPOSS be reconsidered.
So you were aware, were you not, that Jan Holmes had specifically recommended that that earlier recommendation be reconsidered?
Mr Peter Jeram: From reading this, yes, I’m sure, at the time.
Ms Hodge: You have received a copy of the schedule of corrective actions that was circulated in late November – forgive me, at the time you would have received a copy of the schedule of corrective actions?
Mr Peter Jeram: Yes.
Ms Hodge: That’s right. For the benefit of the transcript, that document bears the reference FUJ00079783.
You also received a copy of the revised schedule in May 2000; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: Have you refreshed your memory of those documents –
Mr Peter Jeram: Yes.
Ms Hodge: – from the copies provided to you?
Mr Peter Jeram: Yes.
Ms Hodge: Please could WITN04600104 be shown on the screen, please. So this is version 2.0, so the version dated 10 May 2000 and we can see you there named on the distribution list.
If we could please scroll down to page 10 of the schedule – forgive me, page 9 please. Under the heading “Report Observation/Recommendation” we can see reference to the recommendation to reconsider the redesign and rewrite of EPOSS, that’s right?
Mr Peter Jeram: Yes.
Ms Hodge: You are not the owner of that action and you are not named as one of the management team members. That’s right, isn’t it?
Mr Peter Jeram: Correct.
Ms Hodge: You were, however, involved in its resolution, were you not?
Mr Peter Jeram: Yes.
Ms Hodge: We know from the entries in the agreed actions column, so the second from the right, that Terry Austin had, on 15 November, requested that the recommendation to redesign and rewrite the EPOSS application be closed, having concluded that it would be difficult to justify the case for rewriting it. You were presumably aware of that, were you?
Mr Peter Jeram: Yes.
Ms Hodge: At the bottom of the page we can see that Mr Austin proposed continuing to monitor the PinICL stack for the next few months to assess whether or not it was necessary to re-evaluate that decision.
Mr Peter Jeram: Yes.
Ms Hodge: Were you aware of that at the time?
Mr Peter Jeram: Yes.
Ms Hodge: So, essentially, subject to what the PinICL stack showed, the recommendation was either going to be closed or indeed taken further?
Mr Peter Jeram: Yes.
Ms Hodge: If we scroll on, please, to page 10, we can see two entries dated 8 December 1999. One of these appears to relate to you, that’s the second of those entries. The first reads:
“JH requested statistics on fixes delivered to live from RM. Also informed TPA that requires agreement of MJBC before this can be closed.”
We understand the reference to JH to be Jan Holmes, the author of the report.
Mr Peter Jeram: Yes.
Ms Hodge: Does that sound correct?
Mr Peter Jeram: Yes.
Ms Hodge: He had requested statistics on fixes. This is presumably a reference to statistics from release management; is that right, “RM”?
Mr Peter Jeram: I was wondering if it meant that, but it could do, yes.
Ms Hodge: That would be statistics related to software fixes delivered to the live system; is that right?
Mr Peter Jeram: Correct, or into the testing phases ready to go to the live system.
Ms Hodge: We can see there that Jan Holmes has informed Terry Austin that his instruction to close the recommendation in fact requires the agreement of the programme director, Mike Coombs.
Mr Peter Jeram: Yes.
Ms Hodge: That was correct, isn’t it? The second entry, dated 8 December reads “MJBC”, which we know is Mike Coombs:
“… Confirmed that unless RM statistics contradicted reports provided by PJ the recommendation could be closed.”
So we know, on the one hand, there’s this request for statistics from release management, but there’s also a reference here to reports provided by you. Can you help us, please?
Mr Peter Jeram: Yes, it would be the same data, so I would have given input on PinICLs that had been closed or been addressed and this is asking for confirmation that release management agree with the data that I had.
Ms Hodge: Can I just clarify, you were, of course, the release project manager, why might it be that there were contradictions between your reports and the statistics held by release management?
Mr Peter Jeram: No, there wouldn’t be.
Ms Hodge: Right.
Mr Peter Jeram: He has asked for – I think, in that, Mike is asking for – formalising release management providing the data that matches.
Ms Hodge: Okay, and that’s to satisfy him –
Mr Peter Jeram: Yes.
Ms Hodge: – that it is, indeed, proper to close that action?
Mr Peter Jeram: Yes.
Ms Hodge: There appears then to be a gap of approximately four months and we see the next entry is dated 7 April 2000; can you see that?
Mr Peter Jeram: Yes.
Ms Hodge: In the meantime, concerns were raised with you in early January 2000 about an increasing number of PinICLs, cash account misbalances and reconciliation errors, were they not?
Mr Peter Jeram: I don’t remember. I’m not saying it didn’t happen, but I don’t remember.
Ms Hodge: I’m referring to emails dated early January 2000, which bear the reference FUJ00079332. Please could that be shown on the screen.
The author of this email is a Duncan MacDonald. Was he one of the technical experts on whom you relied to bring technical issues to your attention?
Mr Peter Jeram: I don’t remember Duncan.
Ms Hodge: The email is addressed to you. We can see at the top there it is dated 4 January 2000, I believe –
Mr Peter Jeram: Yes.
Ms Hodge: – although quite possibly it could be 1 April. I’m conscious that some emails have the month first and the day second but, be that as it may, whether January or April, we can see here the subject matter of the report is “CI4 Transaction Mode Problems”.
Mr Peter Jeram: Yes.
Ms Hodge: Do you see that?
Mr Peter Jeram: Yes.
Ms Hodge: I understand CI4 was the name of a software release relating to the EPOSS application which was later introduced –
Mr Peter Jeram: I don’t know if it was just EPOSS, but I think it was an increment forward to the core release. I think it was something like that.
Ms Hodge: So it related to functionality affecting EPOSS and other components of the system?
Mr Peter Jeram: Potentially, yes.
Ms Hodge: I think it follows that CI4 had not been released into the live estate at this stage; is that correct?
Mr Peter Jeram: I would read that –
Ms Hodge: Taking this to be January or indeed April?
Mr Peter Jeram: Because it is talking about end to end systems, so I think it’s having problems in its testing.
Ms Hodge: We’re in the testing phase?
Mr Peter Jeram: Yes.
Ms Hodge: CI4 was part of the larger release known as the Core System Release Plus; is that correct, to your recollection?
Mr Peter Jeram: I don’t know whether CI4 is part of that or whether CI4 was an increment to the release before that, which –
Ms Hodge: The CSR?
Mr Peter Jeram: Yes, because I think CSR+ didn’t happen until quite a bit later.
Ms Hodge: I think that was rolled out in the course of 2000, CSR+.
Mr Peter Jeram: Was it? Okay.
Ms Hodge: I believe. We can check. Be that as it may, we’re dealing here with issues identified in software testing and this email reads:
“We are getting an increasing number of PinICLs on the end-to-end system handling of the new CI4 transaction modes …”
These are described in brackets as “PT”, “NAD”, “RIAD” and “ROAD”:
“… leading to cash account misbalances and reconciliation errors. These PinICLs are generally being batted about between the different areas.
“I suggest a workshop is set up, led by either Requirements or EPOSS, to present the current end-to-end solution, identify the problem areas and then agree the necessary changes to achieve a consistent solution. This may involve having to get clarification of requirements from POCL.
“If anyone can think of a better approach or that there isn’t a problem please say so.”
Do you recall whether the proposal to set up a workshop, whether or not that proposal was taken up?
Mr Peter Jeram: I would have thought so. It’s quite an obvious thing to be suggesting and there obviously were problems that needed to get together and work out what to do.
Ms Hodge: Given that this software release CI4 related, at least in part, to the EPOSS application and there were known to be PinICLs causing cash account misbalances and reconciliation errors, did this email cause you to consider whether or not the outstanding recommendation to redesign and rewrite the EPOSS application ought to be re-evaluated?
Mr Peter Jeram: I think following the workshop and seeing what’s happening in the testing, what’s being found, would have led into that decision about how bad it was.
Ms Hodge: Did this increasing number of PinICLs and the cash account misbalances it was causing not, in itself, call into question the earlier decision of Terry Austin to close the action?
Mr Peter Jeram: Depending on what was found through reviewing this then that might have ultimately led to that.
Ms Hodge: Pathway’s concerns about the quality and stability of the Horizon software were issues of which you continued to have oversight in the spring of 2000, were they not?
Mr Peter Jeram: I think it was about this time that I changed my role. I was asked to look after the development teams. There were some other challenges and we put a corrective action plan in place on certain areas. I’m not sure whether I continued at that point to still be the sort of release project liaison or not.
Ms Hodge: We can see you did have some oversight of these issues. You have been provided with a copy of ICL Pathway’s consolidated risk register –
Mr Peter Jeram: Yes.
Ms Hodge: – covering the period of approximately May 1998 to May 2000; is that correct?
Mr Peter Jeram: Mm-hm.
Ms Hodge: For the benefit of the transcript that document bears the reference FUJ00077884. Apologies, we’re just verifying a reference.
(Pause)
Ms Hodge: Sir, I wonder if you would mind if we take a short five-minute break to see if we can enable the document to be shown on the screen?
Sir Wyn Williams: Well, of course. Just before we do that, Mr Blake and I have been in email communication about tomorrow and am I right in thinking now that the witnesses scheduled for tomorrow are either not going to give oral evidence or be called at some future time?
Ms Hodge: Sir, that’s correct. Mr Jeram is our last witness for this week and I certainly would hope –
Sir Wyn Williams: All right, fine. I just wanted to say that publicly, as quickly as possible, so that anybody listening would know that. So at the end of this afternoon’s session we won’t be convening tomorrow, we will be convening next Tuesday?
Ms Hodge: Thank you, sir, yes.
Sir Wyn Williams: Fine, thank you. Let me know when you’re ready.
Ms Hodge: Thank you.
(2.13 pm)
(Short Break)
(2.23 pm)
Ms Hodge: Sir, thank you for the additional time. We have managed to display the document to which I was referring a short time ago. This is a copy of the consolidated risk register produced by ICL Pathway in the period May 1998 to May 2000. What you can see, I hope, on the screen is page 4 of that risk register where two entries are recorded.
The first bears the reference 00_25. That was a risk raised in February 2000 of which Terry Austin was the owner.
Under the heading “Risk Summary”, it is described as a maintainability – forgive me, that’s the second of the two entries.
The first is 00_38, also raised in February 2000 and of which Terry Austin was also the risk owner. It’s a risk in the area of development, which bears the title “Maintenance activity”. For the benefit of the witness, if we could scroll, please, to the right-hand side, there are some further columns. At column N, Mr Jeram, we see that you were the mitigation owner of that risk; is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: Thank you. Please could we scroll back so we can see column G, which contains the “Risk Description” and we can see that provides that:
“Maintenance effort over the life of the contract exceeds the planned levels. Analysis of call information and user problems necessitates research effort, diverting resources from development or PinICL support work; hence PinICL stacks remain high. Still developing on aged platforms. Skilled and experienced staff increasingly being lost through attrition; no longer able to retain with prospect of developing new applications.”
Then under the column H, we can see a description of the “Risk Impact”, and that provides:
“Cannot retain experienced staff.
“Cannot attract quality people – availability.
“Increased personnel costs in development as staff are retained for maintenance.
“Increased product support costs.”
The following three columns contain an assessment of the probability, the impact and then the factor of those risks. The probability of that risk occurring, that’s in relation to maintenance activity, is recorded as being 3. Do you recall that, Mr Jeram?
Mr Peter Jeram: I can read it, yes.
Ms Hodge: The score of 3, as I understand it, reflected a probability of 30 to 60 per cent of that risk occurring –
Mr Peter Jeram: Yes.
Ms Hodge: – is that right?
Mr Peter Jeram: Yes, I’m looking at the front page, yes, agreed.
Ms Hodge: The impact of the risk, we can see, was scored as 4 and that reflected a “major change for approved costs, quality, timescale of some activity, which would cause serious delay”; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: That gives a factor score of 12. We can see that in column K. Under column L we have a description of the “Mitigation Action” for this particular risk. Thank you. That reads:
“8th June Investment Strategy Board – need positive decisions on future opportunities; POCL need to move away from ‘move to the right’ culture to realise new business opportunities …
“Need to reconsider contingency plan retaining maintenance team for bespoke software with further developments untaken by Large Projects.”
Can you assist us, please, as the mitigation owner, with that entry?
Mr Peter Jeram: I think that mitigation is particularly towards the losing of the skilled resources that understood the products. The point that’s being raised in the risk description is that, if those skilled people are just – I’m not saying PinICL fixing wasn’t important. If they’re just PinICL fixing, then they might want to move on from supporting Pathway. So that was about what were the next opportunities that would keep people interested in doing what they’re doing today, because there’s something different in the future.
Ms Hodge: The concern being expressed here, as raised in February 2000, was that the PinICL stack remained high. That’s what gave rise to this risk of attrition, is it, in relation to staff?
Mr Peter Jeram: Yes, I mean, there was obviously CR+ taking place and there were service increments so there will be continual testing and continual new PinICLs coming along.
Ms Hodge: Just for the benefit of the transcript, by “maintenance activity”, would I be right to understand that what this document is referring to is the identification and rectification of PinICLs, bugs, errors and defects in the system, as and when –
Mr Peter Jeram: Yes, it could be those that were raised at the end of testing cycles that were allowed to be there when we went live, or they could be coming from new changes being made, or they could be coming from the live service.
Ms Hodge: So a number of different domains –
Mr Peter Jeram: Yes.
Ms Hodge: – creating pressures on your maintenance team?
Mr Peter Jeram: Yes.
Ms Hodge: The second entry we see there at row 17, dated February 2000, is described as “Maintainability”. It bears the heading “Maintainability” and is described in column G as a risk relating to the quality of software. It states:
“Products have grown organically so product stability is not assured.”
The risk impact reads that there is “Increased costs, operational system failures and reputation”. Presumably that’s damage to the reputation of ICL Pathway; is that right?
Mr Peter Jeram: Yes, and to the reputation of the system.
Ms Hodge: The probability of that risk is assessed as 2, which I understand to mean that it was considered to bear a 10 to 30 per cent likelihood of eventuating; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: It also carries the impact factor of 4, the same as the maintenance activity risk. The mitigation action is simply to monitor issues in the live estate; is that right?
Mr Peter Jeram: That’s what it says, yes.
Ms Hodge: Do you recall your ownership of that mitigation at the time?
Mr Peter Jeram: Well, it’s not really much of a mitigation really, from what I’m reading there. I mean, parallel with this, we had a corrective action plan that was taking place that was a sort of – quite a big exercise that was in place, going back over designs and software quality and things and sorting out across the estate the supportability for the longer-term of the Horizon product.
Ms Hodge: Forgive me, can you assist us, by a “corrective action plan” are you referring to the plan that we have seen already or are you suggesting that there was another specific plan in place?
Mr Peter Jeram: It came about from the audit, from the development audit, that we put a corrective action plan in place because there was discussion about – at one point of what it was going to cost to do it, which wasn’t a problem.
Ms Hodge: Cost to do what, sorry?
Mr Peter Jeram: To do the work, the extra work.
Ms Hodge: This is to say the maintenance work or the redesign?
Mr Peter Jeram: No, to do improving the documentation of the product that was being maintained.
Ms Hodge: Sorry, I don’t entirely follow. Improving the documentation of the product? Are we referring here to design documentation or are we –
Mr Peter Jeram: Design and test scripts for retesting, et cetera.
Ms Hodge: And to which product are you referring?
Mr Peter Jeram: A number of different products, I think.
Ms Hodge: Right. Forgive me, what we’re dealing with here, I think, is issues in relation to the quality of the software.
Mr Peter Jeram: Yes.
Ms Hodge: We don’t see any reference to a corrective action plan here, do we?
Mr Peter Jeram: No. It’s only that I know that that would have been happening around the same sort of time, but I’m surprised it’s not there as a mitigation. Sorry, that’s my point.
Ms Hodge: Right. What emerges, I think, from reviewing this risk register is that there remained significant concerns in the spring of 2000 about the quality of the Horizon software; is that fair?
Mr Peter Jeram: Yes.
Ms Hodge: And about the ability of ICL Pathway effectively to maintain Horizon in the live estate?
Mr Peter Jeram: That was the risks that were being recorded, yes. There was the risk of that, whereas we can see the probability that – the view was it was a reasonably high probability.
Ms Hodge: I would like to return at this point, please, to the schedule of corrective actions which we reviewed a short time ago. That’s the document that bears the reference number WITN04600104, at page 10, please. Thank you. Forgive me, internal – it says page 11.
There are three remaining entries in the right-hand column, the first of these is dated 7 April, by which I assume we are now into the year 2000. That reads:
“Email to MJBC [Mike Coombs], TPA [Terry Austin] & PJ [that would be you] …”
Mr Peter Jeram: Yes.
Ms Hodge: “… providing details of RM EPOSS fixes to live.”
So release management EPOSS fixes to live:
“Asked for confirmation that matched PJ reports. If does then will close.”
So it appears to show that Jan Holmes has obtained details of the RM, the release management, EPOSS fixes to live and is seeking confirmation that these are matching your reports; is that correct?
Mr Peter Jeram: Yes. I’m surprised it’s four months later than the original entry, but if it’s the same – it feels a bit out of date by then. But, yes, that’s what it’s implying, yes. I assume it’s an updated position by the end of – the beginning of April but …
Ms Hodge: That is dated 7 April. I think the document to which I referred you a short time ago, which I initially thought was dated January, was in fact dated 1 April. Perhaps if we could just pull that up again to ensure we have an accurate record on the transcript. That is FUJ00079332.
If I can just refer you back to the date –
Mr Peter Jeram: Yes.
Ms Hodge: – this is recorded as sent 4/1/2000. Having looked at some of the other ICL Pathway emails, it appears that they bear first the month, then the day and the year. I don’t know if you can assist us with that, whether you think that is correct, that this would have been early April?
Mr Peter Jeram: I don’t remember the dates being in American format but I’m happy to accept – I have seen some that obviously are in American format.
Ms Hodge: Thank you. Can we return, please, to the schedule of corrective actions, which is WITN04600104. As you say, some time has elapsed between the last entry on 8 December and this further entry on 7 April. The next entry, dated almost a month later, is 3 May and this is to record that a reminder email was sent to the above, by which I understand to mean to Mike Coombs, Terry Austin and to you, seeking early response, chased on the same day. Does that assist you at all in your recollection of the progress of this action?
Mr Peter Jeram: No, but looking it through, it seems that, for whatever reason, it took Jan longer to get his data to confirm what I had said back in December. I presume the email was asking for Mike to confirm the position and then Jan has had to chase it again. I’m guessing that’s what happened.
Ms Hodge: Do you recall what your reports were at this stage, in relation to the volume of EPOSS fixes to the live estate?
Mr Peter Jeram: I don’t know but it would have been based on – as I said, it would have been the same data as release management would provide. It would have been the data around what PinICLs had been raised, were open and what had been closed.
Ms Hodge: Bearing in mind the references we saw in the risk assessment – the risk register to PinICLs being high –
Mr Peter Jeram: Yes.
Ms Hodge: – and to the concerns that were raised with you about the increasing number of PinICLs in CI4, is it fair to assume that you are likely to have been reporting that the PinICLs remained high at this stage?
Mr Peter Jeram: If I had reported again at that stage – I don’t know that I reported since the December position. I mean, there’s multiple versions of the product so that CI4 recommendation is for a later development stage of it, as opposed to PinICLs from earlier stages, so they’re all going to start overlapping to a certain extent.
Ms Hodge: But is it your overall recollection PinICLs remained high at this time, in the spring of 2000?
Mr Peter Jeram: I think volumes of PinICLs continued for quite a while.
Ms Hodge: By volumes, you mean?
Mr Peter Jeram: Numbers being raised.
Ms Hodge: High volumes?
Mr Peter Jeram: Yes.
Ms Hodge: The final entry we have is dated 10 May. This records a response received from Mike Coombs, it reads:
“As discussed this should be closed. Effectively as a management team we have accepted the ongoing cost of maintenance rather than the cost of a rewrite. Rewrites of the product will only be considered if we need to reopen the code to introduce significant changes in functionality. We will continue to monitor the code quality (based on product defects) as we progress through the final passes of testing and the introduction of the modified CI4 codeset into live usage in the network. PJ [that’s presumably a reference to you, Mr Jeram] can we make sure that this is specifically covered in our reviews of the B&TC test cycles.”
And the action is recorded as closed.
On 10 May. Just pausing there, the reference to the cost of maintenance – and we have already discussed what we understand maintenance to be – but it is effectively the cost of continuing to rectify bugs, errors and defects –
Mr Peter Jeram: Yes.
Ms Hodge: – in the live estate and in testing.
Can you explain, please, the reference at the very end to making sure that this is covered in our reviews of B&TC test cycles?
Mr Peter Jeram: Yes, B&TC is the – at this stage it went through development testing into system testing and then B&TC, which I think was something like business and technical conformance test, something like that. So what this is ensuring is that the reviews are that, that we’re monitoring what the PinICL position is coming out of those test cycles.
Ms Hodge: So, as at 10 May, the position has ultimately been taken not to redesign or rewrite the EPOSS code; that’s correct?
Mr Peter Jeram: Yes.
Ms Hodge: It was a decision of the management team. As release project manager were you part of that team?
Mr Peter Jeram: I don’t know what the team here would be, but if it’s the management team, no; but if it’s a conversation about what we’re doing with development then I would have probably joined that conversation with Terry and Mike.
Ms Hodge: Do you recall having input into the decision that was ultimately taken in relation to the closure of this action?
Mr Peter Jeram: No.
Ms Hodge: Now, we know that it had been brought to your attention that there were increasing numbers of PinICLs, cash account, misbalances and reconciliations in the CI4 testing.
Mr Peter Jeram: Yes.
Ms Hodge: Were those matters which you had brought to the attention of your senior managers at the time that this decision was taken?
Mr Peter Jeram: Yes, it would have all been – it would have been known.
Ms Hodge: When you say “it would have been known”?
Mr Peter Jeram: Well, because they would have known about what’s happening with CI4 at this point. If there were problems in the testing at the beginning of April then it would have been known by Terry and Mike.
Ms Hodge: So, at the time of taking this decision, it’s your evidence that Terry Austin and Mike Coombs would have been well aware of those concerns which had been brought to your attention?
Mr Peter Jeram: They would have been known – well, I don’t know what it was like at the end of – or the beginning of May, a month after Duncan raised that email, I don’t know what happened during that month, whether after the reviews, et cetera, how it improved. But that’s probably why they’re saying “Let’s monitor this in the B&TC”.
Ms Hodge: This decision was taken, notwithstanding the fact that there were serious concerns about ICL Pathway’s ability effectively to maintain Horizon in the live estate?
Mr Peter Jeram: There were concerns raised in the risk register, yes.
Ms Hodge: Did you consider at the time that this was the right decision to take?
Mr Peter Jeram: I would say yes because, if I didn’t or hadn’t have, then I would have raised my voice to make a point, so I must have agreed.
Ms Hodge: Were you not concerned that the continued application of software fixes was likely to lead to a further degradation in the quality of the EPOSS code?
Mr Peter Jeram: There was still a very large test team in place that were validating things and continually – well, that’s where the PinICLs were coming from, so continually checking and validating and improving the product, so it wasn’t as if that exercise had stopped.
Ms Hodge: You described that as an improvement of the product, but you were aware, were you not, of a risk of what we call code regression?
Mr Peter Jeram: There’s always that risk, yes.
Ms Hodge: What did you understand by that risk?
Mr Peter Jeram: That in changing a product – in an ideal world, once you put something into a live estate this never happens, you would never change it, right? Whenever you change it there is a chance that you will have a code regression in it, that’s why you continue your testing.
Ms Hodge: So it doesn’t follow necessarily that, albeit you might improve certain aspects of the software, you might cause yet further problems to arise elsewhere?
Mr Peter Jeram: Yes and that’s why you do regression testing.
Ms Hodge: Was the likely consequence of code degradation caused by further software fixes, was that not likely to cause yet further problems, such as cash account imbalances and reconciliation errors?
Mr Peter Jeram: There is that risk but that’s why you have the regression testing.
Ms Hodge: Having taken the decision not to redesign and rewrite the EPOSS code, ICL Pathway continued to apply fixes to the code as and when they were detected, did they not?
Mr Peter Jeram: Yes, or as and when they were enhancing it for other reasons.
Ms Hodge: Do you recall whether the number of PinICLs and fixes being applied in the summer and autumn of 2000 remained high or not?
Mr Peter Jeram: No, sorry, I don’t.
Ms Hodge: We have a copy of the release note for the Core System Release Plus, which was produced in October 2000 and to which you appear to have contributed; is that right?
Mr Peter Jeram: Mm-hm.
Ms Hodge: This document bears the reference FUJ00119319, please. This document is dated 24 October 2000. It’s the version 1 of release note for the Core System Release Plus. It provides a definition of the Core System Release Plus for Post Office Counters Limited. Can you please just briefly explain the purpose of a release note of this type?
Mr Peter Jeram: It’s to record and share the contents of the release, so whether that is change requests or PinICLs. CSR+ would have been introducing new functionality, I’m sure.
Ms Hodge: At appendix B, the note contains a known problem register in which known problems in the release and any fixes which have been applied were documented; is that correct?
Mr Peter Jeram: I’m sure it’s correct.
Ms Hodge: You have been provided with a copy of this document, have you not?
Mr Peter Jeram: Yes, I just don’t know what particular part – I can look at appendix B, if you like, to answer your question?
Ms Hodge: No, not at all. We will pull it up. It’s at page 34, please. This is what’s known as the known problem register.
Mr Peter Jeram: Yes.
Ms Hodge: It runs from page 34 to page 39. I’m sure you can help us but I think we can see the specific PinICL references in the far left-hand column.
Mr Peter Jeram: Mm-hm.
Ms Hodge: So their reference numbers –
Mr Peter Jeram: Yes.
Ms Hodge: – and a short summary description in the next column. Then ICL Pathway’s business impact assessment of the PinICL; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: Can you explain the grading, please, for that? We see A, B and C.
Mr Peter Jeram: I don’t know, but I can give you a view. I think A would be that it was a major impact, B of lesser impact and C a low impact.
Ms Hodge: We then see in the next column the business impact on the Post Office Network. We see a record of its status on 10 October and, in the final column, we see commentaries, including whether or not a fix had been applied, whether a fault has been found –
Mr Peter Jeram: Yes.
Ms Hodge: – you follow that?
Mr Peter Jeram: Yes.
Ms Hodge: We, of course, don’t know a great deal about each and every one of these PinICLs and their summary descriptions won’t assist us in that regard, but we can see several relating to the EPOSS – we see quite a number relating to the EPOSS product, or identified as such. Do you accept that, having reviewed this document?
Mr Peter Jeram: Yes.
Ms Hodge: If we could scroll down a little bit, please. Some of these are rated A, so very serious. We see 41673, the “CSR+/EPOSS: OW Sales report negative volume”, categorised A by ICL Pathway and we can see it has been closed, a fix having been applied.
If we go on to the next page, please, we can see a further PinICL at 45573, relating to “Office Balancing Barred”. Presumably, this is another issue in EPOSS if it’s to do with office balancing; is that a fair inference?
Mr Peter Jeram: These, of course, could be problems introduced into CSR+ code and then found during the testing of the CSR+ code.
Ms Hodge: Indeed, in theory, I’m not suggesting – because, of course, this is a release note. This is prior to the release of CSR+ into the live estate –
Mr Peter Jeram: Yes.
Ms Hodge: – is that correct?
Mr Peter Jeram: Yes.
Ms Hodge: So PinICLs discovered during testing, software fixes have been applied and, on the basis of that, the software is deemed to be fit to be released into the live estate?
Mr Peter Jeram: Well, that would be the discussion. Whether the PinICLs that aren’t fixed, whether that’s an acceptable position to go into live.
Ms Hodge: So there are a number of others. I won’t take you through them all, but 47132, the PinICL is “Cannot transfer existing transaction”, again graded A by ICL Pathway, medium severity by Post Office Counters Limited, and closed following another PinICL fix.
Could we scroll down a little bit further, please. Two further PinICLs relating to EPOSS and graded high at 48796 and 488 –
Mr Peter Jeram: I think the concern would be if there was an A here that wasn’t fixed.
Ms Hodge: Well, I think that’s precisely the question, isn’t it, because, of course, by repeatedly applying these software fixes, you were creating a risk of generating yet further faults and defects in the code, were you not?
Mr Peter Jeram: I think that would depend – well, I mean CSR+ would be enhancing the CSR product and bringing in new functionality that the Post Office wanted, so these issues could be in the new functionality in the EPOSS example that’s come in with EPOSS, as opposed to the old product, if you like, the product from CSR.
So, I mean, the problems could come from the development work done for CSR+. I think you’re saying: is that because it wasn’t a stable product in the first place? I don’t think you can draw that necessarily from this.
Ms Hodge: But we do know that, from the emails we have seen, there were quite significant concerns about the number of PinICLs being raised in CI4 –
Mr Peter Jeram: CI4, yes.
Ms Hodge: – or what we believe is one of the releases connected with CSR+?
Mr Peter Jeram: Yes, I don’t know the position of CI4 as against CSR+.
Ms Hodge: Looking back, knowing what we know now, do you consider that deciding not to redesign and rewrite EPOSS, in the face of the advice of ICL Pathway’s technical experts, was the right decision to have made?
Mr Peter Jeram: That’s a very difficult one to answer. Would there have been less problems and less work if it had been redesigned or redeveloped, or would it have created its own problems by starting again? That would be a risk as well and I’m sure that’s the kind of decision that people were making because to start from scratch might introduce its own problems. I think the view at the time was that the problems were in certain areas, as opposed to generally across EPOSS and, therefore, they would concentrate on those areas.
Ms Hodge: Why did you consider that starting again might simply introduce more problems? What led you to that conclusion?
Mr Peter Jeram: It is just the risk of going and redeveloping something right from scratch again.
Ms Hodge: Presumably the purpose of doing that was to ensure that it was done correctly –
Mr Peter Jeram: Yes.
Ms Hodge: – the next time.
Mr Peter Jeram: Which I presume people thought they were doing the first time, but yes.
Ms Hodge: But that didn’t seem to be borne out by the very high number of PinICLs and the advice of the technical experts –
Mr Peter Jeram: Yes.
Ms Hodge: – at the time, was it?
Mr Peter Jeram: I think there were two views, I believe, and the decision was made assessing those two views.
Ms Hodge: Can you help us, what do you mean by “there were two views”?
Mr Peter Jeram: Because I believe some people thought that it could be fixed in the areas that needed fixing and improving and others that felt that the whole thing should maybe be rewritten.
Ms Hodge: You have concluded to yet another view, which is that, by rewriting it, you might create just as many problems. From where did that view stem? Can you help us?
Mr Peter Jeram: It’s just a sort of feeling I got from the conversations at the time. When Terry was leading on deciding which way to go, those were the kind of decisions or discussions that were taking place.
Ms Hodge: I wonder, Mr Jeram, if we could return, please, to your witness statement at WITN04180100.
Sir Wyn Williams: At some stage, Ms Hodge, there’s probably a need for a break, I’m guessing, but you choose your moment, all right.
Ms Hodge: Sir, I have two more questions for the witness.
Sir Wyn Williams: Oh, well, I’m sorry, carry on, please.
Ms Hodge: I propose we take a break at that stage and –
Sir Wyn Williams: Yes, of course.
Ms Hodge: – then permit others to ask questions.
Mr Jeram, you have taken us to your evidence at paragraph 15 on page 4, please. I think you accept that this paragraph does not give a full account of what you knew at the time of issues relating to EPOSS and the cash account; is that right?
Mr Peter Jeram: Yes.
Ms Hodge: I wonder if you could please explain to me why that is.
Mr Peter Jeram: I had forgotten my involvement in that area.
Ms Hodge: I asked you a short time ago about your reference to formal audits at paragraph 26. Do you recall that?
Mr Peter Jeram: Yes.
Ms Hodge: You said that you asked to be shown some of those audits?
Mr Peter Jeram: Yes.
Ms Hodge: Was the CSR+ development audit one of the audits which was shown to you at the time you prepared this statement?
Mr Peter Jeram: I don’t think it was. I might be wrong. I don’t think it was. There was a warehouse audit and something else. I don’t remember, sorry.
Ms Hodge: Looking at what you said at paragraph 26 on page 8, under the heading “Fitness for Purpose”, your statement reads:
“ICL Pathway continually reviewed its work to make improvements for future releases, this would have included formal internal audits, although I do not recall any of these audits specifically. Internal auditing prior to the national rollout was owned by Martyn Bennett, Head of Quality Management, and by his responsible Internal Audit Manager, Jan Holmes.”
This doesn’t, does it, give a true and fair reflection of your knowledge at the time from the internal audits which were shown to you, does it?
Mr Peter Jeram: At the time –
Ms Hodge: Forgive me, “at the time” being at the time of your involvement in the Horizon project.
Mr Peter Jeram: True. So I knew about the audits when I was on Horizon, yes. This is saying I didn’t recall any of them in remembering them as part of the statement, yes?
Ms Hodge: What you are saying, in effect, is that you simply had no recollection at the time of writing your witness statement of the very serious concerns that were raised about the accounting integrity and the quality of code and the maintainability of the Horizon System.
Mr Peter Jeram: Yes.
Ms Hodge: Is that your evidence?
Mr Peter Jeram: Yes.
Ms Hodge: Thank you. I have no further questions.
Sir Wyn Williams: Right. Sorry for the interruption. So do we have questions from other legal representatives?
Mr Jacobs: Sir, I have one question. It’s probably going to take about three minutes, if that assists.
Sir Wyn Williams: And Ms Page?
Ms Page: There may be slightly more than that from me, if I can just have a few minutes to have a look at my –
Sir Wyn Williams: Well, I will tell you –
Can I address you, Mr Jeram. Would you prefer to have a short break now and allow everybody to gather their thoughts, or would you prefer to carry on until the end, on the assumption that the end is no more than about ten minutes away?
Mr Peter Jeram: I’m going to ask Ms Page. I think she would prefer me to have a break so – no?
Ms Page: No, that’s fine.
Mr Peter Jeram: Then can we continue, please.
Sir Wyn Williams: Fine. Let’s go to the end then.
Questioned by Mr Jacobs
Mr Jacobs: Good afternoon, Mr Jeram. I ask questions on behalf of 153 Core Participants who are subpostmasters and I am instructed by Howe & Co.
Can we turn to paragraph 16 of your statement. That’s WITN04180100. This is paragraph 16, which is on page 4 of 9. This is in relation to the cash account and you say that, prior to Horizon, subpostmasters used a paper accounting system. You say that Post Office took a decision which wasn’t taken quickly that there should be no paper cash account; is that right?
Mr Peter Jeram: Yes.
Mr Jacobs: Now, the effect of this decision is that subpostmasters were prevented from checking their records against allegations of shortfalls. They didn’t have the paper system –
Mr Peter Jeram: Yes.
Mr Jacobs: – and the Horizon System didn’t permit them to do that. The question I have for you is: was there any discussion between you, as development and later programme director, and Post Office on this issue?
Mr Peter Jeram: No.
Mr Jacobs: Were you aware of the issue at the time?
Mr Peter Jeram: The issue of?
Mr Jacobs: That the ability of subpostmasters to have records that they could check and interrogate was going to be taken away from them in the Horizon System?
Mr Peter Jeram: No.
Mr Jacobs: I’m asking you these questions because you have referred to it in your statement.
Mr Peter Jeram: Of course.
Mr Jacobs: The final question then is: do you agree from what you have said about the Post Office decision-making process, and that this decision was a decision that Post Office made, that the ability of subpostmasters to check their records was deliberately designed out of the Horizon System?
Mr Peter Jeram: Yes. Obviously moving to an automated system at some point, you would move away from the paper side and I think the papers had to be sent in to TIP and TIP had to process them and – or whatever, but yes, that decision took that away.
Mr Jacobs: Okay. Well, thank you. I don’t have any further questions for you, unless I’m asked to ask you anything else.
No, I’m not. Thank you.
Questioned by Ms Page
Ms Page: Flora Page, representing a number of the subpostmasters.
You became programme director in 2001?
Mr Peter Jeram: Yes, the end of 2001 when Mike Coombs unfortunately was taken ill.
Ms Page: So presumably that meant there wasn’t much of a crossover.
Mr Peter Jeram: No.
Ms Page: But you knew the programme pretty well already, didn’t you?
Mr Peter Jeram: Yes.
Ms Page: And when you took over, what was the line of report between you and the SSC?
Mr Peter Jeram: SSC?
Ms Page: Yes.
Mr Peter Jeram: That’s –
Ms Page: Third line support.
Mr Peter Jeram: – service management.
Ms Page: Sorry?
Mr Peter Jeram: In service management?
Ms Page: Third line support.
Mr Peter Jeram: Okay. None.
Ms Page: So how would you have had any control over what they did?
Mr Peter Jeram: I wouldn’t have had.
Ms Page: You wouldn’t have had? Why not?
Mr Peter Jeram: Because they’re managing the live estate. The programme is managing future change, not what’s in live estate, so I don’t think service management reported to the programme director.
Ms Page: So how would problems in the live estate get communicated to your programme?
Mr Peter Jeram: Through the raising of PinICLs.
Ms Page: And how would your programme come to know about PinICLs?
Mr Peter Jeram: So they would have been routed – my team, development team, would have been fourth line, so when there was something that was felt by the second and third that it required an investigation and maybe a change into software, then that would be routed through to the development teams.
Ms Page: So the fourth line were under your command, as it were?
Mr Peter Jeram: Yes.
Ms Page: Who was that person, who was the link? Who would have reported to you from fourth line?
Mr Peter Jeram: Okay, when I was programme director – so the development director would have managed the development teams and there would have been a number of those different teams and they would have had resolution groups, so the PinICLs would have been sent to those different resolution groups.
Ms Page: Right, but how would you have ensured that problems that were arising in the live estate didn’t continue into the future programme?
Mr Peter Jeram: So following the route, let’s say the problem raised through live through Post Office, investigation from third line believes it’s a software fault, that goes to a fourth line team to resolve, which would have been a resolution into the live estate and we used to use something called a clone. They would then clone that PinICL, a copy of that PinICL, into the version of software they were then working on for the next release. So the fix would be applied into both.
Ms Page: You knew, didn’t you, that throughout the year 2000 Acceptance Incident 376 had been a live issue, something that needed to continue to be monitored?
Mr Peter Jeram: Yes.
Ms Page: So what did you do to ensure that the programme going forward would be alert to and able to continue to monitor what was going on with cash accounts, reconciliations, AI376 generally?
Mr Peter Jeram: I think that would have been through the incidents that came from the live service.
Ms Page: So the route that you have just described?
Mr Peter Jeram: Yes.
Ms Page: Are you aware of that working? I mean can you be sure that AI376 continued to be monitored and fed through to your team for the future?
Mr Peter Jeram: I believe the process worked.
Ms Page: What makes you say that?
Mr Peter Jeram: I mean, if there were bigger problems than just a flow of PinICLs in the live estate I would expect that to have been brought to the attention of the management team that I was then part of, via the service director.
Ms Page: Given what we know now and the fact that there were continuing problems, how can you be confident that this was working?
Mr Peter Jeram: I don’t know how I can be confident. That was the process we had in place that we believed worked.
Ms Page: Were you in any way involved with making sure that if POCL wanted to pursue postmasters, audit trails were made available to them?
Mr Peter Jeram: No.
Ms Page: In 2001 when you took over, who do you think would have been?
Mr Peter Jeram: It would have been managed by the service group.
Ms Page: And who was that?
Mr Peter Jeram: They were under Steve Muchow.
Ms Page: Thank you.
Sir Wyn Williams: Is that it, Ms Page?
Ms Page: Those are my questions, thank you.
Sir Wyn Williams: That’s it, Ms Hodge?
Ms Hodge: That’s right. Thank you, sir. The witness can be released.
Sir Wyn Williams: So thank you very much, Mr Jeram, for coming to the Inquiry to answer all the questions put to you.
As I have said, we will now adjourn these hearings until 10.00 on Tuesday.
Ms Hodge: Thank you, sir. Good afternoon.
Mr Peter Jeram: Thank you.
(3.10 pm)
(The Inquiry adjourned until 10.00 am on Tuesday, 15 November 2022)