16 November 2022 – David McDonnell and Jan Holmes
Hide video Show video
(10.00 am)
Sir Wyn Williams: Mr Beer, before we get going with the evidence, I have a number of announcements that I would like to make. I’ve got a carefully prepared script so that I don’t get it wrong. So let me read it out.
During the course of his oral submissions at the commencement of this phase of the Inquiry, Mr Stein asked me to consider permitting recognised legal representatives to make what he described as “short oral submissions”, after the evidence had been heard at the close of each phase of the Inquiry. Having given the request some thought, as this phase of the Inquiry has been in progress, I have decided that, if any recognised legal representative wishes to make short oral submissions about the evidence heard in Phase 2, they may do so on Friday 2 December. Each legal representative who wishes to speak will be limited in time to 45 minutes.
As an alternative, recognised legal representatives may prefer to make written submissions about the evidence heard in Phase 2. If any legal representative opts to make written submissions, those submissions should be no greater than 25 pages in length and should be filed with the Inquiry by 12 noon on Wednesday, 7 December.
I stress that this decision should not be taken as setting any kind of precedent for what I might permit in relation to the remaining phases of the Inquiry.
Second announcement: I am now in a position to provide further details as to the hearings relating to Phase 3 of the Inquiry. These hearings will take place between 10 January 2023 and 20 January 2023 and between 14 February 2023 and 10 March. All these hearings will be held here at the IDRC, and the sittings will take place Tuesday to Friday in each of those weeks.
I have decided that there is no need to begin the Phase 3 hearings in the week commencing Monday 12 December, as I had previously announced. By delaying the start of Phase 3 until January next year, there will be additional time for appropriate disclosure to the Inquiry pursuant to Rule 9 requests and further time for the Inquiry team to provide disclosure of relevant documents to the recognised legal representatives or Core Participants.
The break in the sittings between 20 January and 14 February has been caused by the need to secure suitable accommodation for the hearings, but the fact of the break will also allow for disclosure of documents to and by the Inquiry to proceed in a way which does not impose too great a strain upon those involved in this arduous process.
Next, it is now possible to make a reasonable prediction as to the likely timing of Phases 4, 5, 6 and 7. My current intention is that oral evidence will be taken in relation to Phases 4 and 5 in the period between 2 May 2023 and 31 July 2023. Oral evidence in relation to Phase 6 will commence in September 2023, and it is anticipated that oral evidence in relation to Phases 6 and 7 will be completed before the end of 2023.
For the avoidance of any doubt, there will be no hearings in August 2023, and I will endeavour to be as flexible as is reasonably possible in relation to the fixing of dates for hearings in September 2023.
I hope it will now be obvious to everyone that the volume of documentation of relevance to the Inquiry is such that there is a need for breaks between the various phases, and the need for a reasonably substantial break between the end of Phase 3 and the beginning of Phase 4 is, I hope, self-evident.
I would like to confirm my intention to hold a hearing on 8 December 2022 so that I can be updated on the various compensation issues which are still under consideration, and thereafter give consideration to making an interim report to the minister to be laid before Parliament pursuant to the Inquiries act 2005.
Finally, I should say that I am monitoring the issue of disclosure of documents, as I promised I would, and I will not hesitate to convene a hearing before or after Christmas if I consider that there are issues related to disclosure which would benefit from oral submissions.
So thank you all very much and over to you, Mr Beer.
Mr Beer: Thank you, sir. May I call David McDonnell, please.
David McDonnell
DAVID McDONNELL (affirmed).
Questioned by Mr Beer
Mr Beer: Thank you, Mr McDonnell. As you know, my name is Jason Beer and I ask questions on behalf of the Inquiry. Can you tell us your full name, please.
David McDonnell: David McDonnell.
Mr Beer: May I thank you on behalf of the Inquiry for providing a witness statement and coming to the Inquiry today to give evidence to assist us with our investigation. We thank you.
You should have in front of you a hard copy of your witness statement which, excluding the exhibits to it and the index to the exhibits to it, is 18 pages in length, and it should be dated 3 November 2022.
David McDonnell: Correct.
Mr Beer: Is that your signature on page 18?
David McDonnell: Yes, it is.
Mr Beer: For the transcript, that’s WITN00620100 at page 18. Can I just invite you please to turn to page 15 and paragraph 47. Two lines from the end, a name is given. It says:
“By a staffer from Bracknell called Nick Lawson.”
Do you wish to make a correction to the surname of that individual?
David McDonnell: Correct.
Mr Beer: What’s the correct surname?
David McDonnell: It should read Nick Lawman.
Mr Beer: L-A-W-M-A-N?
David McDonnell: Correct.
Mr Beer: Similarly, in paragraph 55 on page 17 four lines in it reads:
“From Bracknell, Nick Lawson.”
Same correction?
David McDonnell: Correct.
Mr Beer: With those two corrections, are the contents of the statement true to the best of your knowledge and belief?
David McDonnell: Yes, they are.
Mr Beer: A copy of that statement will be uploaded to the Inquiry’s website. So I’m not going to ask you about every aspect of it; do you understand?
David McDonnell: Yes.
Mr Beer: Can you tell us something about your background at qualifications, please. So to start with your qualifications.
David McDonnell: I achieved a Bachelor of Science honours degree in Computer Science from Sheffield Polytechnic, graduating in 1987. During that period, I spent a year-and-a-half on a sandwich placement at ICL Bracknell and, subsequently, after graduating a further year-and-a-half or so again back at ICL Bracknell working on their software development system.
Mr Beer: So I think you joined ICL Pathway in Easter ‘98; would that be right?
David McDonnell: Yes.
Mr Beer: You worked at ICL Pathway for a couple of years; is that right?
David McDonnell: Yes.
Mr Beer: What was your job title when you initially started?
David McDonnell: I was brought in by Chris Humphries and placed into the EPOSS counter development team as the deputy to Steve Warwick, deputy development manager.
Mr Beer: So deputy development manager. If you had a business card, that’s what it would say on it?
David McDonnell: Yes.
Mr Beer: As you have just said, you were in a team working on the EPOSS counter component of the Horizon System; is that right?
David McDonnell: Correct.
Mr Beer: In terms of the structure of the team, was Steve Warwick the project manager?
David McDonnell: Yes, he was.
Mr Beer: Was he a full-time ICL employee or was he a contractor?
David McDonnell: He was a contractor.
Mr Beer: Do you know to whom he reported?
David McDonnell: He reported to Terry Austin, but actually in reality quite a lot of people in the building. He had a very extensive knowledge of how the Post Office worked and tended to talk to quite a lot of people. But he reported directly to Terry Austin.
Mr Beer: Just expand on that, that he had quite a good knowledge of other people within the building.
David McDonnell: Yes, his network was quite formidable really. He knew pretty much everybody in the building and had a lot of knowledge of the history of the project and, to be honest, what he didn’t know about how the Post Office worked, you know, didn’t exist. He was very knowledgeable about how the systems worked.
Mr Beer: In terms of Terry Austin, he was the program manager; is that correct?
David McDonnell: Yes.
Mr Beer: That was his formal report?
David McDonnell: Yes.
Mr Beer: Chris Humphries, what was his position on the team, please?
David McDonnell: Chris Humphries was – I’m not sure exactly what formal title would be, but he was under Terry Austin, and he managed projects such the EPOSS counter in more of a deputy program manager or departmental management role. He didn’t get involved technically day to day, but he managed the kind of programmers as deputy program manager, I would say, under Terry.
Mr Beer: There were, I think you say in your statement, eight or so developers on this team; is that right?
David McDonnell: Yes.
Mr Beer: Was there a hierarchy amongst the developers?
David McDonnell: Not really. It was quite a flat eight, I would say. There was various degrees of experience and capability in the team, but there was no formal structure as such. It was flat under Steve Warwick when I arrived.
Mr Beer: You tell us something in your statement about Chris Humphries expressing deep concerns over the quality of the EPOS team and of the code being produced at interview.
David McDonnell: Yes, it came out as early as that during my interview with him. He didn’t go into specifics or names. He did allude to quite a few concerns and the reason why I was being brought in.
Mr Beer: What was the reason for you being brought in?
David McDonnell: He needed to bolster the team with some experienced, technical, formally qualified ability, was my understanding.
Mr Beer: You said – I’m sorry –
David McDonnell: Sorry, go on.
I think he suspected that Steve Warwick, as knowledgeable as he was about the Post Office, wasn’t technically or formally qualified, and that is probably what led to his concerns about why he needed to bring somebody with a little bit more formal experience in to the team.
Mr Beer: What was your understanding of what Steve Warwick was qualified in?
David McDonnell: At interview, I didn’t hear anything as such, but very soon after arriving it was very clear that Steve’s immense knowledge of the Post Office tended to override anybody’s opinion or anybody’s views, because he could very easily engineer a discussion, with such wealth of knowledge on the subject, at a business level, that it was very difficult to argue or contribute to such a conversation on a technical level. So almost everybody deferred to him in terms of his knowledge of the business processes of how the Post Office worked.
Mr Beer: Was Jan Holmes a member of the team?
David McDonnell: No.
Mr Beer: What, to your understanding, at this time on recruitment in Easter ‘98 was Jan Holmes’ role?
David McDonnell: I don’t think – although I new Jan personally from being in the building, Jan didn’t have an involvement directly in the development team until the task-force really. I know that he was doing work around the project and in his audit capacity, but not specifically day to day in the EPOS team at that time.
Mr Beer: You say in your statement that there was one technical assistant, Brian Orzell. What do you mean by technical assistant?
David McDonnell: Brian’s role was to take care of – he was kind of the NT expert on the team who took care of administrative privileges and user roles and all of the kind of NT stuff and not – he wasn’t a developer. He did assist in facilitating extractions of message stores or any technical questions that people had or any of the PinICLs that came through which were more NT orientated, such as blue screens, things like that, Brian would get involved in diagnosing those.
Mr Beer: Thank you. What were your duties and responsibilities on the EPOS team?
David McDonnell: When I initially started, it was to run the team technically on a day-to-day basis, the development team.
Mr Beer: By run, can you explain what it meant, what you mean by that. Do you mean as a manager, HR professional or –
David McDonnell: No, pretty much, within a matter of a few days, Steve Warwick was so busy working on future requirements or business liaison, whatever it was he was doing on a kind of business-process level, I very quickly took over the day to day management of the developers, the code deliveries, the PinICL management, the change release management software, things like that. So I worked with the developers on software developer, PinICL resolution and trying to – well, very quickly got into the stride of things we needed to improve.
Mr Beer: You told us that in the interview Mr Humphries expressed deep concerns over the quality of the EPOS team and the code being produced. What concerns did Mr Humphries raise in respect of the quality of the EPOS team?
David McDonnell: Chris already had a feeling that some of the guys weren’t up to it. They weren’t sufficiently experienced or capable or had the ability to kind of take on the work that was required to get this thing built and over the line. He didn’t mention any names specifically at the interview, but he had a pretty good handle on the fact that things weren’t right in the counter team.
Mr Beer: You say in your statement that within days of starting it became very obvious to you that several members of the development team were not capable of producing professional code.
David McDonnell: Yes.
Mr Beer: How many members of the team of eight?
David McDonnell: Out of the eight, I would say two were pretty good, very capable; another two were kind of mediocre but we could work with them, they could contribute positively; and there was probably three or four who just weren’t up to it.
Mr Beer: You used the phrase “not capable of producing professional code”. Does that accurately describe the level of your concern?
David McDonnell: Yes.
Mr Beer: You say that in interview Mr Humphries raised deep concerns in respect of the quality of the code that was being produced. Can you remember anything more than that?
David McDonnell: His take on it then was not at a level where he would have inspected the code or read any of the code per se. He was responding to basically the lack of quality which was being signalled from the test team with the number of PinICLs being raised.
Mr Beer: You say that within days of you starting it became obvious to you that there were no development standards or methodology. What do you mean by that, development standards or methodology?
David McDonnell: Well, a project such as that – well, any kind of software development project, there should be a framework of how the team work. It should start with the design documents. That’s the target of what you are trying to deliver; that’s what you are building against. They weren’t in evidence. I know that there had been some documents that were reverse engineered, but they were irrelevant and out of date, and they weren’t even in the building when I got there. I had to ask for them.
Methodology-wise, back then it was mostly kind of a PRINCE2 kind of.
Mr Beer: Just explain in a couple of sentences to those that might be listening with PRINCE or PRINCE2 methodology, what that is.
David McDonnell: It’s basically just a set of rules that you follow, that you must have a certain structure of documentation provided. When you write code against them you should follow a certain sequence of steps, such as code reviews or coding standards, and there should be a coding standards document which specifically states how to go about programming in a language such as Visual Basic, how to use variable names, what naming conventions you use for your function calls, technical level stuff. The reason for a document such as that is that, when you have 8 or 10 or 20 people coding, they all produce a similar-looking code which has the same naming conventions so that, if the next person has to pick it up, it looks very similar and it reads the same.
Mr Beer: You say in your statement that it became obvious to you that there were no coding practices. Is that different from the things you have just described?
David McDonnell: Yes, similar. I think practices is a more generic term which would cover testing, unit testing, so testing your own code after you’ve produced it, producing documentation to show what you have done, that kind of stuff.
Mr Beer: You say, thirdly, that there were no peer reviews. Why do you consider that to be a problem or a difficulty?
David McDonnell: Well, I think it’s human nature that, if anybody’s left to their own devices, they kind of start to drift towards making life comfortable for themselves, which might result in people writing code which does not adhere to the standards you have agreed and, by implementing peer reviews, that is enforcing the fact that you have to show a colleague or a peer what you’ve written, why you’ve written it in a certain way, and just basically explain it to them, read it through with them, and, if everybody’s in agreement that is a correct interpretation of the design, it adheres to the coding standards that’s agreed and you have carried that correct testing, et cetera, et cetera, then that usually enforces compliance to the expectations of the team.
Mr Beer: Would peer reviews in coding be a normal or unusual practice?
David McDonnell: It’s standard. It’s expected.
Mr Beer: You say that it became obvious to you that there were no unit-testing standards. Can you explain firstly what you mean by unit-testing standards.
David McDonnell: Yes. The nomenclature of testing gets a little bit fuzzy depending on who you ask, but a coder should test his own code, and he should test it not just in isolation but also within the kind of landscape of all of the other modules that live around it. So he will be expected to carry out that unit test on an environment which is – although it’s in a development environment, it is representative of the latest release that’s out in the testing-hardware environment.
He should test that code himself thoroughly to a degree that it’s not as intensive as when it goes into the test cycle, but it’s pretty thoroughly tested for the code that he’s added or changed, but also in kind of a doughnut shape around it, to make sure that he hasn’t affected anything else that that code might interface with.
Mr Beer: Thank you. You say there were no design specifications in place, but what do you mean by those?
David McDonnell: I didn’t see any design documents for the EPOSS counter when I got there, to say, “This is the architecture of the EPOSS counter. These are the modules.” I think a high-level design existed, but for the actual modules which comprised the EPOSS counter to say, “This is the cash account report, this is the selling a stamp bit”, whatever it is, those modules should have had a low-level design document which detailed how it was built, and what the functionality inside it was, and what the interfaces were. Those documents I never saw.
Mr Beer: Who out of Post Office and ICL would be responsible for drawing those up?
David McDonnell: That would be ICL.
Mr Beer: You draw those points together by saying:
“In fact, this team was like the Wild West.”
What did you mean by that?
David McDonnell: Pretty much as it says on the tin. There were no standards in place, there were no design documents. The culture of the development team was – I wouldn’t say it was a holiday camp, but it was free format. There was no structure, no discipline; it was crazy, never seen anything like it.
Mr Beer: When people refer to the Wild West, they sometimes mean people acting in accordance with their own wishes, not according to standards or conventions or rules. Is that what you meant by –
David McDonnell: Yes, I think, if you take that phrase, it probably means lawlessness, and that lawlessness I was trying to refer to was lack of standards, lack of rules, lack of discipline, lack of structure within the development team.
Mr Beer: You say that several of the development team were not capable of producing professional code. Did that impact in your view on the integrity of the EPOS system?
David McDonnell: Fundamentally. If they weren’t capable of demonstrating to me or an auditor or anybody else in the building that they could write a simple piece of code in a professional standard, then I had to ask myself: what have they been writing for the last 12 months or however long they’ve been there? What’s under the bonnet in the system already that they’ve contributed?
Mr Beer: What, if any, concerns did you have as to the impact of the issues that you have just identified on the operation of the system when it was eventually used by subpostmasters?
David McDonnell: Well, we’re here today, aren’t we? I think that’s evidence in itself. It was clear – it was clear that the system, as it was at that time, was never fit for purpose in a real-world environment. Everything I’d seen when I first arrived and everything I’d seen in the short period I was on the counter team told me that you should never be putting this piece of software into a live estate without at least fixing everything that we recommended needed fixing so it was fit for purpose.
Mr Beer: You say in paragraph 9 of your witness statement that it was:
“A company-wide well-known fact that there were several thousand outstanding bugs in the EPOS system.”
We have been told that any system of this size, not just looking at the EPOS system but Horizon generally, is likely to contain a number of bugs. To what extent were the bugs in the EPOS system usual in terms of number and severity or out of the ordinary?
David McDonnell: Well, I think what you should see in a development life-cycle of a project or software is you might get a small number of bugs that make themselves evident to start with, obvious ones, glaring ones, but that should decrease over time, and the severity of them should decrease over time as well.
That’s not what we were seeing. What we were seeing was a constant high level of PinICLs being raised daily, and that number was not diminishing, and they weren’t getting any simpler. They were becoming more and more complex as code was introduced.
You should also never see that volume. That amount of PinICLs – as soon as I saw the number – when you raise a PinICL, it gives you a new number, it’s incremental. When I saw the number of the next PinICL to be raised, I was astonished at how many had already been through the system and closed, for us to get to that number that it was going to offer you next. So that tells you how many’s been raised and closed already, which to me was completely out of kilter with the size of the project or the complexity of it.
So that, along with the fact that the number of PinICLs being raised was not diminishing, tells you that the quality of the code was – you know, something wasn’t right.
Mr Beer: How do you know that this was a company-wide, well-known fact?
David McDonnell: Because it was quite a sociable project in the building. We mixed with all of the other test teams. I had a very good relationship with all of the test teams, and some of the other development teams who were developing things such as APS. If you look that PinICLs that were raised on those parts of the project, they were nowhere near as large in number as they were on the counter and, when you talk to these people, it was a standing joke, “You’re in the EPOS team, good luck”, or … You know, it was bête noir of the building.
Mr Beer: You say in your statement that the EPOS team was the joke of the building; is that right?
David McDonnell: Yes, I think everybody knew, specifically the test team who, when I spoke to those guys, they would make it very clear that the quality of code that was being delivered was to such a bad, poor level that they’re wasting their time testing it, because they knew that it was just broken. They were going to end up raising lots of PinICLs from it. So they’d give a very frank and very honest opinion about the ability of some of the guys, not all of them – some of them were good – in the team, and the quality of the product that that team was producing. So it was a standing joke in the building.
Mr Beer: You say in your statement that this was known up to the highest level, including Fujitsu Japan, because they sent over three coders to perform an audit. Can you recall when that took place?
David McDonnell: I’m afraid I can’t recall whether it was before or after the task-force initiative. It was around about that era. It was probably late summer or maybe afterwards. It may have even been into the following pre-Christmas/post Christmas period.
Mr Beer: So the task-force, we understand, the PinICL Task Force, to have been August and September ‘98?
David McDonnell: Yes.
Mr Beer: And you can’t recall whether the coders sent from Japan were before or after then?
David McDonnell: I can’t, I’m afraid.
Mr Beer: What kind of auditors were they? Were they from Fujitsu or were they external?
David McDonnell: They told me they were – they were Japanese and they were from Fujitsu in Japan. So, as far as I understood, they were Fujitsu employees.
Mr Beer: What did they do?
David McDonnell: They spent maybe two days, maybe three days – I gave them a desk with an EPOS development counter in the team and showed them around little bits, where to find the software, how to get access. I called in a couple of the guys to help talk them through some of the modules they wanted to look at. It was very brief and arm’s length, but they spent two or three days looking at the very lowest level, the code, some of the reference data, how the counter was built, et cetera. It was mostly a code review.
Mr Beer: What was the outcome of that audit by these coders from Japan?
David McDonnell: No idea. They came, they sat, and they went, and they didn’t speak to anybody.
Mr Beer: Was any report to your knowledge produced back to you at least –
David McDonnell: No.
Mr Beer: – of the outcome of the audit of the code?
David McDonnell: No.
Mr Beer: To your knowledge, did Terry Austin have any contact with the three Fujitsu Japan coders?
David McDonnell: Yes. Well, he told me they were coming and I was to facilitate whatever they needed, but that’s the extent to what I know.
Mr Beer: There came a time when a task-force was set up, as it was called, PinICL Task Force can you explain the circumstances in which it came to be set up.
David McDonnell: Yes, I was probably a few months into working in the team. By then I had a pretty good grasp on how much trouble that part of the project was in. I’d spent several months trying to work with Steve and Chris Humphries and Terry to get them to understand where we were and how it was, without making much headway.
It was basically everything was carrying on as normal and I wasn’t really making any progress in improving things that dramatically.
I already at that point had a view on what needed to be done, and it was when Steve Warwick went on three-week vacation that I kind of had the opportunity to speak to Terry more directly without Steve Warwick being there. He fronted that relationship on behalf of the team up to that point. As he was away on vacation, I got the opportunity to kind of talk to him more directly.
But also at that time there was some important test cycles coming up, such as model office releases and things like that, which were quite important to the roll-out schedule of the project and, given that, with the amount of PinICLs that were being raised by the test team, it was an opportunity to use the importance of that delivery with the amount of PinICLs we were experiencing to emphasise to Terry that we need to do something dramatic about this, otherwise we’re going to fail, and that’s when Terry and I – he asked me what I needed, “What do we do, what do you need?” I kind of described what I thought we could do, you know, if I was given the right resources and the correct time, et cetera, et cetera, and he instigated the task-force and gave me carte blanche to pick whoever I needed from the building to join the task-force to help us.
Mr Beer: Did you do that?
David McDonnell: Yes.
Mr Beer: Was a report produced as a result of the work undertaken by the PinICL Task Force?
David McDonnell: Yes, it was.
Mr Beer: Can we look, please, at FUJ00080690. It will come up on the screen for you, Mr McDonnell. You can use the paper copy if you wish. Is this the report that you are talking about?
David McDonnell: Yes, it is.
Mr Beer: Just to introduce it by its abstract first, it says:
“This document reports on the activities of the EPOSS PinICL Task Force which was in place between 19 August and 18 September 1998 to reduce to manageable levels the EPOSS PinICLs outstanding at that time.”
Is that an accurate description in high level summary?
David McDonnell: Yes.
Mr Beer: Just look, please, at the dates of the reports that we’ve got. This is the report that we’ve all been working from to date because it was the only one that we had. Can you see that it’s dated in the top right-hand corner 14 May 2001? It’s said to be version 1 and it carries the reference IA/REP/008.
Can you go over the page, please, Frankie. Can you see on Document History that version 0.1 is said to have been produced on 18 September 1998 and was the initial draft following Task Force completion. So that’s at the end of the period, the date period, described in the abstract, 18 September 1998. Would that accord with your recollection, that it was produced shortly after or at the very end of the Task Force work itself?
David McDonnell: Yes.
Mr Beer: Was it being written as you went along?
David McDonnell: Yes, it was. I mean, we were gathering evidence and understanding as we went, yes.
Mr Beer: If we go back to the first page of this document, please, can we display at the same time FUJ00121098. FUJ00121098. If we crop – thank you.
We can see here a further version of the report that’s been recently disclosed to us. You can see that the title is the same and the abstract is the same. It’s still said to be a draft. The version number is 0.1 which is obviously before by convention, I think, 1.0; is that right?
David McDonnell: Yes.
Mr Beer: The date is said to be 16 February 2000, which obviously wasn’t one of the dates mentioned in the document history when we looked at page 2 of the previous version.
Then, if we can go to the second page of this document, please, the one on the right, we can see version 0.1, which coincidentally is also this version or said to be this version, the one of 16 February 2000 is dated 18 September 1998.
Looking at that information, do you believe that there ought to exist a version dated 18 September 1998 called initial draft following Task Force completion?
David McDonnell: Yes, my understanding is that we 0.X were drafts, and then once it’s formalised it becomes version 1.0. So that date fits, and that’s certainly the document that we contributed to.
Mr Beer: Just one other piece of information on this dating and version issue. Can we look, please, at FUJ00079782. This is a completely different report dated 28 October 1999. You will see what it is from the abstract there. Can we go to the second page, and go down please.
By convention Fujitsu reports list documents that are associated with the document that’s being written. Can you see at item 6 there is a reference to IA/REP/008 which is said to be version 0.3 and dated 29 September 1998 with the correct title Report on EPOSS PinICL Task Force?
David McDonnell: Mm-hm.
Mr Beer: So there ought to be a version 0.3 available –
David McDonnell: Yes.
Mr Beer: Would that be right?
David McDonnell: Yes, I’d say so.
Mr Beer: Can we go back, please, to the first one we were looking at which is the one we’re using because it is version 1.0 which is FUJ00080690. You will see, if we scroll down a little bit, please, that you are one of the co-authors. Is that right, that you co-authored this document?
David McDonnell: Yes.
Mr Beer: What form did that co-authorship take? Did one of you write it and the other one approve it, both of you write it, both of you write bits of it; how did it work?
David McDonnell: I think Jan was the kind of audit expert running with the document, and my contributions would have been made by – I’m not sure I would have opened the document myself and typed it in, or whether I emailed him the text and he posted it into the document – more likely.
Mr Beer: Both of you signed it off; is that right?
David McDonnell: Yes.
Mr Beer: Rather than just one of you?
David McDonnell: Yes.
Mr Beer: So it was your joint work?
David McDonnell: Well, Jan would have come to me and made sure I was happy with the content, and we both agreed to sign it off, yes.
Mr Beer: Looking at the distribution list, can we just run through it and can you explain why it went to each of those people. First Terry Austin.
David McDonnell: Well, Terry’s program manager for the development team. He has the ultimate decision on, or responsibility for the counter code, what needs to be done, and I believe he reported to Martyn Bennett, who obviously needs copying, because there were some quite serious issues inside the document that he should be aware of.
Mr Beer: What level was he within the company?
David McDonnell: I’m not exactly sure. I think he was senior to Terry Austin.
Mr Beer: D McDonnell: why were you getting a copy back?
David McDonnell: I don’t know.
Mr Beer: Library: what was the library; was that a physical library or an electronic library?
David McDonnell: I believe Jan would have copied that so that it was archived, but probably he could answer that better than me.
Mr Beer: Was that a physical library or –
David McDonnell: I believe it was electronic.
Mr Beer: What was the purpose, to your understanding, of Fujitsu having a library of documents like this?
David McDonnell: Days like today, I guess. I think as an auditor it’s probably the audit trail.
Mr Beer: If we look at the approval authorities on page 2, please, firstly, what is an approval authority?
David McDonnell: I’m not sure in that context. I don’t know what that would mean. Whether it just means that the document’s been signed off as accepted by the recipients, I’m not sure.
Mr Beer: Can you now recall what happens if somebody didn’t give their approval?
David McDonnell: No, I can’t. I don’t really understand what that term would result in, what that was for in the document.
Mr Beer: In the earlier version that we’ve got the version 0.1, said to be dated 16 February 2000 – you remember the other one I showed you?
David McDonnell: Mm-hm.
Mr Beer: The second approval authority was Martyn Bennett. Have you got any knowledge of him giving approval for the document?
David McDonnell: No. I think, once that’s gone up and has been released, it would have been by email copy. I never heard any feedback from anybody on that list other than Terry Austin. I certainly didn’t ever have anything back from Martyn Bennett.
Mr Beer: I should just say, for the record, track change comparing the two versions that we’ve got show that the only changes are the addition on this version on 0.1 there of “1.0 14/5/01 raised to version 1. Administrative catch-up” and that change from Mr Bennett to Mr Holmes as well as, obviously, the version number and the date that appears on the top right-hand corner of each page.
Do you know what administrative catch-up meant or means?
David McDonnell: No, I don’t.
Mr Beer: Mr Holmes, as you said, he’s a witness in the Inquiry who is going to give evidence to us later today, worked for Pathway as an audit manager and, in his written evidence at least – the cross-reference needn’t be displayed, WITN04600100 – at paragraph 9F identifies two concerns.
He says that there was a concern about the technical accuracy and structure of the EPOS code when it had been written. Do you agree with that description?
David McDonnell: Yes.
Mr Beer: Did you discuss that with him – Mr Holmes?
David McDonnell: Yes, we will have done during the Task Force period at least, yes.
Mr Beer: Secondly, he says there was a concern which he considered to be the greater of the two concerns which related to the impact of continual changes to existing code to fix problems and/or to insert new functionality into the code. Do you agree with Mr Holmes that that was a concern?
David McDonnell: I do and, in fact, within this document there’s a very good example of that when, during the Task Force, which was supposed to be all about getting the quality under control, they took away some of the resource to force in extra functionality for, I think it was balancing and something else. There’s three parts to it. It’s referred to in the document somewhere. But it was a sizeable piece of development work which was being developed on the fly and shoehorned into the code, right in the middle of the Task Force initiative, where we were trying to stabilise the product, and that’s a typical example of not understanding the problem of where we were at the time and continuing with the same bad behaviour, in my view.
Mr Beer: Those two concerns that Mr Holmes mentions and which you agree with, in your view, would they have had any impact on the integrity of the system, how it operated or how it was operated by subpostmasters?
David McDonnell: Yes, it would. It would result in functional errors, bugs, spurious behaviour.
Mr Beer: Was that a view held by you and others at ICL Pathway at the time?
David McDonnell: Yes, it was. I think it was a belief that was pervasive throughout the building.
Mr Beer: Can we look, please, at page 4 of the report and just read the introduction:
“During the week commencing 17 August the EPOSS Counter PinICL Stack Reduction Team, known as the task Force, was established. The objectives, current workload, composition, outline process and targets were presented to the team on Tuesday 18, with a formal start date of Wednesday 19th August 1998.
“This report presents the outcome of the Task Force activity and identifies factors which prevented the original target (zero or near to zero residual PinICLs) being met.”
Just stopping there, was that the target of the Task Force?
David McDonnell: So this is an interesting point. That is the written kind of objective, the most desired outcome that would be great if we could get the PinICL stack down. My personal view was that we’d never be able to reach zero PinICLs, because we knew that the code was in such a bad state that that would never happen. So I think there was kind of a difference between the expected outcome from people like Terry Austin and the expected outcome from the people who had a technical understanding of what was happening on the ground.
Terry’s ideal world would be that we get back to zero PinICLs and the ship sails on, whereas the people on the ground who actually knew what was happening and the state of the code were expecting the outcome – personally, myself personally, was that hopefully this gives us sufficient evidence to be able to move the project on to a different footing, which would be corrective, such as rewrite the cash account and the various other recommendations that were made. I was under no illusion at all or belief that there would be zero PinICLs that end of this.
Mr Beer: I was going to ask you about that, because this report does not just report on the work of the PinICL Task Force and the reduction in the number of outstanding PinICLs, it takes the opportunity to make a series of significant criticisms on EPOS.
David McDonnell: Correct.
Mr Beer: Was that deliberate?
David McDonnell: That is what I was alluding to. This was an opportunity for the technical people who understood it to get this stuff onto the paper and get it front of some senior people with evidence, to show them what kind of a state we were in and what needed to be done. It wasn’t just about: let’s go and fix a thousand PinICLs and the problem goes away. So this document was used as a vehicle to kind of put that evidence in place and get it in front of somebody.
Mr Beer: The report continues:
“During the course of the Task Force it became clear that there are significant deficiencies in the EPOSS product, its code and design, and these are also presented in this report.”
Did it only become clear during the Task Force that there were significant deficiencies in EPOSS code and design, or did you in fact know about that beforehand?
David McDonnell: We knew about that beforehand.
Mr Beer: I think you are saying that this was an opportunity to make it clear?
David McDonnell: So that had been voiced vociferously throughout the project. Not just myself but the test team had voiced that view to everybody beforehand. But this was a kind of formalisation and a last chance to get that evidence enforced really in documented format.
Mr Beer: “Finally, the report contains recommendations from the authors which we believe should be implemented by the program to address the shortcomings identified.”
I’m going to skip over Scope. I’m going to skip over Management Summary at the foot of that page and go to the page 5, please. You say:
“The EPOSS Task Force was established to address the problem of the escalating number of PinICLs residing in the EPOSS-Dev and Counter-Dev stacks and was planned to operate for the five weeks leading to the MOR3 baseline cut on 18 September.”
Can you remember what the MOR3 baseline cut on 18 September was?
David McDonnell: I think it was model office. I don’t know what the R stands for, but that was one of the key pillars of the acceptance testing plan, and so that had to be successful in order for the acceptance testing to progress.
Mr Beer: “The objective was to reduce the PinICL count to zero or low tens by the cut-off date, and the target that set by dividing the current PinICL count by the number of days available. The paper made no concession towards new PinICLs being raised during the period and assumed that the personnel assigned to the exercise would be available 100 per cent of the time and be 100 per cent effective.
“The position at 1 o’clock on 18 September is that 166 PinICLs have been fixed and closed and 165 remain in …”
Is that “work in progress”?
David McDonnell: Yes.
Mr Beer: “This indicates the Task Force has failed to meet its prime objective.”
Then you say this:
“However, a review of the Task Force period provides an insight into why it was unable to meet its objective. This Management Summary provides an overview of that period and is supported by the main body of the report.”
Can we go to some of the whys, please, rather than looking at the PinICL-reduction exercise and go over the page to page 6, and look at EPOSS documentation which is a bit further down. You say:
“The document suite supporting the EPOSS product code consists of three main elements …”
You set them out.
“All of these were developed by reverse engineering the EPOSS product code at that time.”
Are you saying by that paragraph, those sentences, that the EPOSS product code was reverse engineered, or that the documentation was reverse engineered?
David McDonnell: I believe it’s the documentation it’s referring to.
Mr Beer: If we go forwards, please, to page 16, we can see in the top three paragraphs a reference to those three documents. Can you see that:
“The returned product was then reverse documented and version 3.2 of the EPOSS Functional Specification produced in December ‘97.”
David McDonnell: Yes.
Mr Beer: Then in the next paragraph you say:
“During April ‘98 an EPOSS High-Level Design document” –
You say “reverse engineered”; do you mean reverse documented?
David McDonnell: Yes.
Mr Beer: Similarly, in the next paragraph:
“Corresponding Low-Level Design documents were developed during July ‘98 by ISTL, again reverse engineered” –
But do you mean reverse documented from the code?
David McDonnell: Yes.
Mr Beer: Is there a difficulty with reverse documenting?
David McDonnell: So what they’ve done is they have basically taken the code as it’s written and they’ve produced the design specification which should have produced the code, and they’ve written the design document afterwards to match the code that was already in place. So they are chronologically reversed.
Mr Beer: Is there a problem with that?
David McDonnell: Well, other than the fact it’s the horse before the cart, no. It’s a very simple task of looking at the code to see what it does, and writing a document to say so. It’s never going to be wrong because you’ve read the code and it matches.
Mr Beer: So what you are saying is that one should start with a specification, one should start with a high-level design document, one should start with a low-level design document, and then write product code accordingly, to those specifications and designs, not the other way round?
David McDonnell: Yes.
Mr Beer: Who was doing the reverse documenting here?
David McDonnell: I think they got some technical authors in. They are referring to ISTL, but I can’t remember who that was. We didn’t really have a great deal to do with it in the counter team, because it was – it’s a moot point, that documentation.
Mr Beer: What do you mean, it’s a moot point?
David McDonnell: Well, instead of it contributing to having a design document which specifies how the code works, they’re basically writing a document which mirrors what’s already been done. So to us it was irrelevant.
Mr Beer: Do you know whether Post Office was told about this?
David McDonnell: I wouldn’t know that. But that is indicative of somebody has to – some standards have to be met. Somebody’s going to do an audit to say: are the right documents in place? Well yes, they are now, but they weren’t when the code was written. So it looks good on paper, but that isn’t the design waterfall flow that should have been followed.
Mr Beer: Can we go back to page 7 of the report, please. Under EPOSS code in section 7.2, you say:
“It is clear that senior members of the Task Force are extremely concerned about the quality of code in the EPOSS product.”
Who were the senior members of the Task Force that were extremely concerned?
David McDonnell: I consider the senior members to be myself and Jan, and Jan can speak for himself later, I guess, but there was probably two or three technical people that were brought into the Task Force team who had excellent credentials, and they did some of the low-level analysis as part of the Task Force team, and I guess together, myself with those two or three guys, we all formed the same opinion.
Mr Beer: You used the words “extremely concerned about the quality of the code”. Why were you extremely concerned?
David McDonnell: Well, it’s – it was so bad. It was beyond anything I’ve ever seen. Even in the 25/30 years since that project, I’ve never seen anything like that before. Some of the stuff that we found buried in the code was unbelievable. There was unreachable code. I mean, we pulled out some of the better examples –
Mr Beer: We’re going to come to those.
David McDonnell: That was a small number of examples as to what we found. Just the whole – you could see looking at the code, the way it was written, the different modules, no standards had been followed. It was a mess.
Mr Beer: You say:
“Earlier this year the EPOSS code was re-engineered by Escher, and the expectation is that the work carried out in Boston was to a high standard and of good quality.”
Can you explain that process, what happened there.
David McDonnell: I don’t know. That was before I arrived. I’m not aware as to how much rewriting they did or reverse engineering they did.
Mr Beer: You say:
“Since then many hundreds of PinICL fixes have been applied to code.”
Here are you just referring to the EPOSS code?
David McDonnell: Yes.
Mr Beer: “… and the fear is that code decay will, assuming it hasn’t already, cause the product to become unstable.”
What did you mean by code decay, please?
David McDonnell: Code decay occurs if you have to revisit the code and rewrite it to fix bugs that have been raised. The danger is that you start to – the code that was written with its initial intent starts to diverge away from what should have been a clear specification. The more frequently you do that, the more divergence there is, until in the end the code that you’re left with bears little resemblance to the original design specification.
Mr Beer: You say:
“This presents a situation where there is no guarantee that a PinICL fix or additional functionality can be made without adversely affecting another part of the system.”
David McDonnell: Yes, because what we were seeing in – there was that many PinICLs being raised, and guys had gone in and they might put a two-or-three-line- fix in or a very small correction, but there was that many of them, some of the corrections you couldn’t understand why that correction had been made three or four months ago, for example, because it’s not documented. There was no documentation to show why that particular line had been changed. So somebody might go in and say, “That’s wrong”, and they’d change it back to suit the case that had been written today, without understanding that it now reverses the fix that was made maybe several months ago for a different reason.
Mr Beer: You continue:
“A more worrying concern from the Programme’s perspective should be the reliance on the EPOSS product in its current state as a basis for planning and delivery. … there was relatively little testing that directly impacted … yet more than 200 PinICLs, roughly 50 a week were raised. Immediately following the conclusion of the Task Force, it is intended to re-run System Test Main Pass and various other test streams. While I am confident that the fixes delivered by the Task Force will prove to be reliable, I fully expect the PinICL rate to increase as further testing is carried out.”
The “I” in that sentence, is that you or Mr Holmes?
David McDonnell: I think that sounds like Jan’s paragraph, looking at the wording, but I would agree with it. If you replace it with “we”, I don’t think it would …
Mr Beer: You continue:
“Lack of code reviews in the development and fix process has resulted in poor workmanship and bad code.”
Then you say:
“Four examples are presented [later].”
Can we move on please to page 12 of the report at paragraph 6.2. You refer in this paragraph to:
“… poor quality workmanship from some of the more experienced team members as evidenced by an average 33 per cent reject rate from unit test and a failure of every build due to missing RD or code.dlls.”
Can you elaborate what you mean by “poor quality workmanship” that you here describe.
David McDonnell: Yes. So what was happening was the project, or the EPOSS counter team, had got into such an exhausted state that the culture had become: throw a fix in the code, throw it over the fence at the test team. There was very little control of the release mechanism from the development team into the test team.
So sometimes a lot of this stuff could have been really stupid things such as they’d only partially release the fix. Some of the modules or little bits of software that had to go with it, such as a DLL file or a header file, were missing from the work package that was released over to the test team.
It could be of that nature. It could also have been just the quality of the fix itself was – as I was referring to earlier, they’d fix this bit but that would break it over here. So that was very typical of what we were seeing at the time.
Mr Beer: Thank you. Can we go forwards, please, to page 15 and the bottom half of the page under 7.1.1, under Documentation Suite, you say:
“The EPOSS product was originally developed using RAD …”
What does RAD mean?
David McDonnell: That stands for rapid application development.
Mr Beer: “… techniques as part of the Joint Working Agreement in force during Release 1. This approach carries a number of attendant risks, not least of which is the lack of formal specification.”
Can you explain why the RAD technique carried with it attendant risks, please.
David McDonnell: Yes. So what they have done there as part of the bid process is to use something that was often referred to as rapid prototyping, where you would throw up a skeleton kind of pro forma of what you think it might look like and how it might work, but without much engine room behind it, if you like, and that is very sensibly used for a bid process or a proof of concept or something like that.
What appeared to have happened here is that they used that and progressed it forward into the main code base, rather than actually saying, “Okay, we’ve got a prototype we know what it should look like and what it intends to be for proof of concept. We’re now going to start it from afresh and design this properly using good engineering principles and design processes and start from afresh using the prototype as a model to work from.” But you would never use that code in the real product. You’d start again.
Now –
Mr Beer: Just so I understand it, if you were an architect or a builder building a house, you might want to build a model of it using balsa wood –
David McDonnell: Exactly. You never use the model for the bridge, even if it was one-to-one life size. That’s what they did and, back in the day, there was a design development methodology called rapid prototyping, which was a pre-cursor to what is known today as Agile, which came out of California. But back in those days rapid prototyping was very immature, and it should never have been used on a project of this size/complexity. So they’ve kind of half used that as an excuse to justify why they’ve taken the initial prototype and used it moving forward.
Mr Beer: Can we turn over the page, please, to page 16. Thank you. In paragraph 7.1.2, at the foot of the page, you say that:
“POCL identified three major gaps in the EPOSS product, namely Discounts, Transfers and Stock Unit and Office Balancing – referred to as the three papers – and these were required for implementation into EPOSS.”
Can you remember when these issues, the three papers, were identified by Post Office Counters Limited?
David McDonnell: My first awareness came about the start of the Task Force or shortly before. This is indicative of exactly how Steve Warwick used to work. He was the interface with POCL, and he was the person who would be discussing with them what the business requirements were for the EPOSS counter, and stuff like this used to pop up all over the place. “Oh, by the way we have this; by the way we have that.”
This was introduced at the last minute as a must-have requirement for model-office testing or model-office acceptance. So it’s an example of quite a sizeable piece of work that was being stuffed into the code in a very rushed, last-minute way, right in the middle of a Task Force which was absolutely essential for model-office acceptance.
Mr Beer: In the next paragraph you say:
“A third issue raised by POCL was the manner in which the proposed functionality had been presented in the specification. Whereas version 3.2 described EPOSS on the basis of the ‘accounting cycle’, POCL wanted it to reflect their business processes. The result was that POCL were invited to develop ‘solution proposals’ which if acceptable would be factored into version 3.3 to provide the level of detail requested by POCL. To date some 57 solution proposals have been presented by POCL, although only 6 have been reviewed and passed for inclusion in the specification.”
Can you help us with what the solution proposals related to.
David McDonnell: They were business functionality, as perceived by POCL, of how they wanted the counter to operate.
Mr Beer: Can you remember what happened as a result?
David McDonnell: Well, these were – Steve Warwick would field these with POCL, and he would feed them into the counter development team as requirements, and then the guys would work on that functionality and introduce it to the code. I don’t know – I can’t remember how many of these actually came to full fruition or were developed out.
Mr Beer: You say:
“The final area of difference revolved around the EPOSS issues list.”
What was the EPOSS issues list?
David McDonnell: I believe that was not a risk register but an issues register that was managed between – that’s the kind of interface between ICL and the Post Office, where the list of known issues which had to be managed away or explained before Post Office would accept the product.
Mr Beer: What kind of issues would feature on that list?
David McDonnell: Cash account, missed balancing, blue screening.
Mr Beer: And:
“This was, [it is said], replaced by the ‘Request for Clarification’ process …”
Can you recall why it was replaced by the request for clarification process?
David McDonnell: I don’t know. I wasn’t involved in that.
Mr Beer: And it says:
“To date some requests for clarification have been received by POCL.”
What sort of issues were raised in the 90 or so requests for clarification?
David McDonnell: So those would be – once Steve Warwick had brought the requirement into the development team, it may be that the business analyst or some of the developers might be raising questions on how exactly this is supposed to work, or this piece of functionality was to operate in real life. So those questions would be raised back to POCL via those RFCs.
Mr Beer: Would those matters impact on the integrity and robustness of the system?
David McDonnell: Yes, the more toing and froing – and this generally used to happen after the code development had started. So, if they came back with a different answer to what was expected some, of that code may have to be modified.
Mr Beer: Can we turn to the next page in paragraph 7.3 of the report, so page 17. In the box above the text it says:
“This section has been produced with the assistance of Dave McDonnell [you] and Martin Smith …”
Who was Martin Smith?
David McDonnell: Martin Smith was a developer from another team in a different part of the building, and he was one of the people who was asked to join the Task Force.
Mr Beer: Why was he asked to join the Task Force?
David McDonnell: Because he was probably one of the most capable people in the building in terms of development.
Mr Beer: You say:
“Although parts of the EPOSS code are well written, significant sections are a combination of poor technical design, bad programming and ill thought-out bug fixes. The negative impact of these factors will continue and spread as long as the PinICL fixing culture continues. This is partly due to the nature/size of the bug-fixing task and partly due to the quality and professionalism of certain individuals within the team. The problem is probably best illustrated by examples.”
Then you give some examples.
Can you look at the example, example 1. Can you explain to us what the problem is.
David McDonnell: Yes. Somebody’s written a function here which is called by a part of the code to reverse the sign of an integer or something, and basically it surmounts to: number equals number times minus 1. Why would you write a function to do that? It basically demonstrates two things, really: First of all, a complete lack of understanding of basic mathematics which I think is written below; and, secondly, I just can’t understand why anybody would write that. It’s beyond comprehension.
Mr Beer: You say over the page:
“Whoever wrote this code clearly has no understanding of elementary mathematics or the most basic rules of programming.”
You referred in that paragraph to the quality and professionalism of certain individuals within the team. Were they the individuals that you referred to earlier?
David McDonnell: Yes.
Mr Beer: How might an example, example 1 in your document here, affect subpostmasters on the ground?
David McDonnell: That actual example would function correctly, but the fact that it’s been written tells you that the person who wrote it doesn’t understand. So that’s a red flag or a flare to say, if they thought that that was necessary, what else have they gone and done in the code elsewhere? So that’s evidence that you couldn’t trust the rest of the code.
Mr Beer: Can we look at page 19, please. You give an example of unreachable code. What does unreachable code mean?
David McDonnell: Those three lines would never be executed under any circumstances, because the logic of the code would not – would never fall into that part of the code.
Mr Beer: Unreachable code means that the function will not be carried out?
David McDonnell: Correct.
Mr Beer: Executed?
David McDonnell: Yes.
Mr Beer: In a scheme of sort of mildly poor practice to fundamentally wrong, where does this sit?
David McDonnell: That’s about as bad as it gets. I mean, if that piece of code was actually critical to the cash account or selling a stamp, you’d never be able to achieve the expected outcome in a business sense.
Mr Beer: Example 3 at the foot of the page, “poor workmanship and patchwork PinICL”. I have to admit I didn’t completely understand this one. Can you help to explain it, please.
David McDonnell: Yes. This one’s a little bit more kind of nuanced that somebody’s written this as an “If, then Do”, et cetera, et cetera, when really there’s a very simple “While Do” loop that should have been used which is much, much cleaner and more accurate. I’m not sure why we thought that was a patchwork PinICL. Maybe at the time there were some comments to say that this had been inserted or amended in order to fix a PinICL.
Mr Beer: Hence patchwork?
David McDonnell: Yes.
Mr Beer: Then, lastly before the break, over the page, please, example 4 Hard Coding. You say that that is an example of hard coding which might have been made for a good reason, but there’s no evidence of review to remove. What did you mean by that?
David McDonnell: Well, if you’re trying to fix quite a complex PinICL, it’s sometimes the very quickest route to get there is to hard code the specific example of the bug you’re trying to fix, to get it to work properly, which is why they’ve got a hard-coded date in there, and some of these numbers at the bottom in the middle, sorry, are hard coded. But having understood and resolved the problem, what should happen is at that hard coding should be taken out and it should be either parameterised, which means that you can change it in a header file very easily as text, without having to recompile the code, or in some instances it may be referring to reference data where it should have been rectified in reference data which can be easily passed down to the counter without a software release.
Mr Beer: Thank you very much for that explanation, Mr McDonnell. Sir, that’s an appropriate moment to take a break if it suits you?
Sir Wyn Williams: How long do you think we should take given the need to move smoothly, so to speak?
Mr Beer: We’re on track, sir, to finish by lunchtime. So 15 minutes, please.
Sir Wyn Williams: Mr McDonnell, everybody knows that Mr Holmes is listening to your evidence. I am sure you wouldn’t think of it but, if you do think of it, don’t have a word with him about it in any break, all right?
David McDonnell: Understood.
(11.23 am)
(A short break)
(11.41 am)
Sir Wyn Williams: Yes, Mr Beer.
Mr Beer: Thank you. Mr McDonnell, can we go back, please to FUJ00080690, the report, and the first page of it. You see on the distribution it’s got Mr Austin, Mr Bennett, yourself and the library. You’ve mentioned Chris Humphries and Steve Warwick as being important members of the team. Why was the report not addressed to them?
David McDonnell: I don’t know recall why. Might have been a better idea to do so, a more expansive distribution list, in hindsight.
Mr Beer: You’ve addressed it to two senior people, Mr Austin and Mr Bennett. Can you assist whether there was any view at the time that, if you’d addressed it to Mr Warwick or Mr Humphries, something different might have happened?
David McDonnell: Okay, so I’m going to give my honest kind of reflective view on this, that Steve Warwick was on the other side of the camp, that no rewrite was necessary, everything was fine, ship’s sailing on nice. So he wasn’t going to effect change as a result of this document. In fact, he was probably against the recommendations.
Chris Humphries couldn’t do anything about it. We’d already tried that route, myself and Chris, before the Task Force was initiated and, therefore, we already knew that Chris couldn’t do anything about it under his own initiative. It needed a sign-off and the commitment of Terry Austin, Martyn Bennett. That’s probably why it’s a more limited distribution list.
Mr Beer: I understand, thank you. Can we look, please, at your witness statement WITN00620100 and at page 3 of the witness statement, please. At the foot of the page, paragraph 12 having exhibited the report you say:
“I understood the underlying cause of concerns to be that the bid had been won using a prototype which had then been further developed upon instead of starting afresh properly.”
You have explained that to us already.
“Additionally there had been a lack of formalised signed-off designs, a lack of discipline, a lack of professional qualifications in key positions …”
You have explained that to us already. Then you say:
“ … a total disengagement of the chief architect Gareth Jenkins …”
Just stopping there, did you know Gareth Jenkins at the time of your work from Easter ‘98 onwards?
David McDonnell: Yes.
Mr Beer: What was the extent of your work contact with him?
David McDonnell: Almost zero. So my understanding was that Gareth worked alongside another chief architect under Alan Ward. Now Gareth Jenkins’ responsibility was specifically to the EPOSS counter system. As the chief architect, I would have expected him to be much more involved in overseeing a lot of the previous coding standards and methodologies and things like that, and certainly the design documents that we’ve referred to earlier.
I was quite surprised that he was based in Bracknell. I had to dig him out rather than him coming to the building to visit the team. Unless I made a specific effort to try and talk to him about something, he was just not present on the project.
Mr Beer: Is that what you mean by total disengagement?
David McDonnell: Yes.
Mr Beer: If we go forwards, please, to page 4 of your witness statement at the foot of the page, at paragraph 16, you say that you:
“… requested access to a copy of the design specification and all existing documentation for the EPOS system. I did this because it’s the starting point of all engineering – it sets out what you are trying to build. It is also important in managing and meeting the client’s expectations and demands. Some of this documentation was located and I was given access to that, but it was totally out of date.”
You say that the specification is the starting point of engineering because it sets out what you’re trying to build. What would a design specification look like?
David McDonnell: There’s different levels of design specs. It will be a high-level design which was much more of a bridge between the business and the technical development team, and that would have more references in it to business functionality, how to sell a dog licence or a stamp or whatever, and how that translates into the counter system.
Then below that you have maybe one, maybe more, low-level designs which usually break out into functional subject matter, which have a much more in-depth specification of exactly how that module should work, what data it should use, what interactions it should have, which APIs it should use, et cetera, et cetera. There may even be much lower-level specifications underneath that if required.
Mr Beer: You say you were given access to some of the documentation. Can you remember what documentation you were given access to?
David McDonnell: So, when I first got there, took a quick look round. “Okay, where’s the design specs? Let’s go back to basics and see what we’re supposed to be developing. What does the bridge we’re supposed to be beginning look like? Where’s the design?”
I did manage to locate some of them and I believe those were the re-engineered – reverse engineered documents that we referred to earlier. It certainly wasn’t a comprehensive set of documentation for the counter, and it wasn’t in a tiered architecture with a high-level design, et cetera, which I would have expected.
Mr Beer: What was the effect of the material that you were given being out of date?
David McDonnell: Well, it was worthless and irrelevant. The only purpose it could possibly have served would be to satisfy an audit at a high level, to say, “Are there design documents in place, and have they been followed”, and somebody might have been using that as acceptance criteria at a different level.
Mr Beer: Can we go forward to paragraph 18 of your witness statement, please. You say:
“So far as I was aware, ICL Pathway had, in fact, dived in and progressed the prototype into development with no structures or process around it. This approach is fatal in a large project with several integrations.”
Why is the approach fatal in a large project with several integrations?
David McDonnell: So that was historical to before my time there. So I’ve derived that comment from what people told me when I arrived on site that, because the development was quite well underway when arrived, they told me that that was the historical nature of how they’d arrived at where we are today, at that point.
Jumping in – so to answer your question, jumping in and progressing the rapid prototype is pretty much the answer I told you earlier about taking the model and turning it into the real thing.
Mr Beer: To use my analogy, the building on balsa wood?
David McDonnell: Yes. Just to elaborate on that, the larger and more complex integrations that a project such as that might have, the more chance there are – obviously increases the chance of errors being made across interfaces and up and down the software stack.
Mr Beer: You continue:
“The client …”
That’s Post Office; is that right?
David McDonnell: Yes.
Mr Beer: “… was allowed to scope creep and retroactively add to and change the requirements which was accommodated by Steve Warwick.”
What effect did that have on the development or operation of the Horizon System?
David McDonnell: Well, it’s kind of several-fold really. First of all, the business requirements were not clearly laid out, which led to the fact that the high-level/low-level designs were never properly produced against the business requirement, and what you tend to find on projects where these things aren’t in place is that the client then has freedom to either change the initial business request that they made, or morph it, or even ask for extra functionality, and you end up in an argument as to whether that was included in the first request or whether it’s a supplementary request.
Because these were being facilitated quite a lot by Steve in his conversations with POCL, we were getting development requests being shoehorned into the counter team, right to the last minute, and an example of that this one which I referred you to earlier about the – which was the three-part papers which shoehorned – the software development was shoehorned in whilst the Task Force was underway. So you get late requests, changing requests, scope creep.
Mr Beer: Can we go forward, please, to paragraph 19 which is over the page. You say, on completing your initial assessment of the system, you concluded that:
“ … 70 per cent of it could be saved, fixed and tidied up, 20 per cent needed a lot more work but could be kept, but that the critical Cash Account module was beyond repair and must be rewritten. There was a layer of design missing from the EPOS system which would ensure only validated messages could be written to the message store. There should have been an Application Programming Interface [that’s the API that you referred to a moment ago] between the code and the message store which ensured that only correct and validated messages could be written to the Riposte message store instead of the freestyle that was currently allowed. The freestyle was like having a graffiti wall instead of a library with the Dewey system. Instead of each module reading and writing messages to the message store in a freestyle manner, they should only talk to thing Application Programming Interface which would only accept and reply to strictly controlled, documented and audited read/write messages and it itself would read and write the messages to and from the message store.”
Is it right that that’s what led you to the cash account module being beyond repair?
David McDonnell: Partly, yes. I mean, this is a much more fundamental point regarding the design of the counter system. So the Riposte message store and that message replication system underneath worked quite well, and that did have an API which was exposed to modules that were built on top of it, to allow them to read and write messages to the message store. An example of such a module might be the cash account, for example, or, if somebody sells a stamp, the reference data’s read up and the messages are the transaction that are written down.
Now, what we were saying here was that there should have been an EPOS-specific API in between those two which restricted read-and-write access to and from the message store, and only allowed messages to be written down to the Riposte API which conformed to – the message contents conformed to the standards which were defining in the data dictionary document.
What that would have done is controlled the contents of the messages being written to the message store, and prevented people writing stuff into it which was not conformant to the agreed vocabulary or reference data or anything else that was defined.
Mr Beer: What was the effect, if any, of not rewriting?
David McDonnell: The fact that this was missing allowed the developers a freestyle approach that, if they went into to add some extra functionality or to fix a PinICL, they could all of a sudden introduce a new message, a new message type, to the message store to make life easy for themselves to resolve the problem that they were trying to fix or code that they were trying to implement.
Because they did not conform to any standard, if another module came along that had to read that message or depended upon the contents of that message in order to maybe accumulate a correct cash account, for example, if the original developer deviated from an agreed set of data inside the message, that module may not pick it up, or it may read the wrong field, or it may accumulate something incorrectly or it may miss the message altogether.
So that standard compliance being missing could lead to any outcome you care to imagine.
Mr Beer: What do you mean by any outcome you care to imagine?
David McDonnell: Well, we used to see it all the time with certain products. If you sold a particular product combined with something else, for example, when the cash account accumulated, sometimes that product sale, the transaction was completely missing, because it didn’t recognise the message that the sale transaction had put into the message store, or it may have got confused or used a different product code or something. It could be anything. It could manifest itself in any imaginable way really.
Mr Beer: So the risk would be, if I took an example that you sold six months’ road tax, the subpostmaster would enter that transaction on their counter and, because of the problem that you identified here, that transaction would be entirely absent?
David McDonnell: It could be, or it could be – yes, I mean, if someone’s gone in to fix a PinICL and they’ve introduced a new message which slightly deviates from the other kind of road tax or something like that, or he’s typed something in wrong or misunderstood it, then that particular thread of transactions wouldn’t be collected as part of the accumulation.
Mr Beer: In my example it would be if in combination you sold six months’ road tax and bought a book of stamps?
David McDonnell: Something like that, yes, yes.
Mr Beer: Can you remember now any hard examples of this? I appreciate that’s some ask …
David McDonnell: I can. I can’t give you a hard, firm example, but this used to happen. All the time. The common request that was made during problem resolution say, for example, model-office testing was underway and the test team raised a bug, a PinICL, probably one of the first requests – you couldn’t really do anything without a copy of the message store. So the request would go to Brian Orzell, “Please you get me the message store for that particular period of time.” The message store would be provided back, and then it was a case of wading through the tens of thousands of messages inside the message store, to try and follow the thread of that sales transaction, and then try to interpret why it wasn’t accumulated or why it was represented incorrectly, and that was a very manual, labour-intensive job that was also quite time critical because, if you didn’t do it quickly, the message store would move on rapidly, and you were unable to reproduce the problem because the particular set of circumstances under which that PinICL happened has disappeared now, it’s moved on. So that’s why it was critical to have a defined set of messages.
The other thing it speaks to as well is that there were no diagnostic tools for the developers to be able to dig into the message store and say, “Right, we sold the car tax on Tuesday at 2 pm. Show me the thread of transactions which resulted from the sale and show me how they were accumulated into the cash account.” That diagnostic tool was missing.
Mr Beer: Did the absence of the Application Programming Interface lead to a significant risk to the integrity of the transactions undertaken by subpostmasters?
David McDonnell: Absolutely it did. In my view, it was one of the biggest shortfallings of the counter design. You had no control whatsoever over what was getting written to the message store. It was the code and the PinICL fixing which decided – it was coming down like confetti rather than being channelled.
Mr Beer: Can we go forward to paragraph 21 of your statement, please. You say:
“It was also possible for anyone to read and write anything into a message and post it to the message store outside of the EPOS modules.”
Can you explain what you mean by that.
David McDonnell: Yes, I must qualify that by saying that you must have two sets of permission to write to the message store. The first is you must have the correct NT user permissions and, secondly, within Riposte there was also user permissioning as well. But, if you had those, it was very straightforward to use a command-line interface such as Riposte, put message or something with a text string which had a message in it that you could use to insert into the message store, and in fact that method was used quite frequently to correct cash account mis-balances. They would –
Mr Beer: Who is the “they” in that?
David McDonnell: Well, we used to get it in a lot of the test environments used to produce cash-account mis-balances, and the fix was: if it’s £2,000 over, you send a minus £2,000 message into the message store, and it will cancel it out, and that would allow the test cycle to continue. So –
Mr Beer: Would that be without addressing the fundamental –
David McDonnell: Well, that’s –
Mr Beer: – or underlying issue?
David McDonnell: Correct, that’s just fixing the message balancing, not the code that caused it.
Mr Beer: You have qualified 21, to the extent that it should read anyone with the two permissions that you have mentioned. How large a cohort of people would the “anyone” be, with that qualification?
David McDonnell: In practice I think it was usually Steve Warwick who generated the message into a batch file, and that would be released through the normal package-release system, release-management system, and then that would be dropped on to the counter as a piece of code and executed as a batch file.
So it would be usually Steve that did that; maybe Brian Orzell occasionally helped him.
Mr Beer: What about outside of the EPOSS team, because in the next sentence you say:
“This was a technique used on occasion by the support team …”
David McDonnell: So my understanding was, later on in the project when I wasn’t on the counter position, but I was led to believe that that technique of correcting mis-balances was being used by the support team, but that would have come from somebody in the EPOSS counter team. They would have constructed the command, wrapped it into a batch file and passed it to the support team for them to put through Tivoli as a code release to correct the error.
Mr Beer: Where did you get that understanding from?
David McDonnell: Just from talking to the guys on the team.
Mr Beer: You say in and paragraph 22:
“I reported my conclusions to the following people:
“Steve Warwick, who ducked and dived and swerved the issue.”
What do you mean by that?
David McDonnell: Steve was very pro: “We’ve done everything right, there’s nothing wrong.” He was in that camp. “The code’s in good enough condition to be able to Go Live.” So by then I was just making noise to all of these people. They’d already decided that it wasn’t going to get wholly or partly rewritten, and that’s an important phrase which we should come to in a minute.
But Steve was firmly in the camp of: We’re not rewriting it; it’s okay as it is.
Chris Humphries agreed. He could see the problem and he did try very hard to effect change, but he was under Terry Austin, and he didn’t on his own have the political sway to be able to persuade the higher echelons to, you know, bite the bullet and rewrite the cash account.
Mr Beer: So he actually refused, Mr Austin, for the cash account to be rewritten?
David McDonnell: Yes, Terry Austin did. He disagreed with me and –
Mr Beer: You say that Gareth Jenkins denied the issues point blank and ran off to hide in Bracknell and avoided contact with the team. In what way did he deny the issues?
David McDonnell: Well, we managed to get Gareth down to the counter team I think twice that I can recollect, and we tried to engage him in the conversation about the missing API, which he was very defensive of and said, “No, there’s nothing wrong with it as it is.”
I also tried to engage him to get him to lend his political design weight behind the argument that at least the cash account should be rewritten, if not the whole thing, and I was unable to get him to engage on our side to lend his persuasive weight to persuade Terry Austin to rewrite the cash account.
When we started having conversations like that, that’s when he kind of became evasive, certainly with me. I was never able to get him to come back down on site again after that.
Mr Beer: In the list of things that you thought needed to be done, was total or partial rewrite of the cash account at the top of it?
David McDonnell: So that phrase that we’ve used in the document that Jan and I authored, we recommended that it should be rewritten in part or in whole. This is a key phrase because, although it’s not as clearly written in the document as it could have or should have been maybe, those conversations and emails were certainly taking place within the project with the likes of Chris Humphries, the test team, Terry Austin, everybody, that “in part” meant we should at least rewrite the cash account, because in my view this was primarily a financial accounting system at the end of the day. If the system blue-screened or you couldn’t sell a stamp on a Tuesday, or whatever it was, that’s an inconvenience, but at least financially it’s correct.
So my recommendation and those of the senior members which we spoke about earlier was the cash account must be rewritten at least. That’s the “in part” part of that phrase.
Mr Beer: Was that in answer to my question at the top of your list?
David McDonnell: Yes, that was the number 1 thing that needed to be done.
Mr Beer: Can we go forward to paragraph 41, please, of your witness statement which is on page 13. In paragraph 41, you say:
“I have observed several witness testimonies referring to the proposed ‘rewrite’ as a big deal, a big job that could potentially introduce more problems than it would fix. This was not necessarily true and indicates either a basic misunderstanding of how the EPOSS system was built or even potentially suggests an attempt to obfuscate the issue. The EPOSS system was modular and what the other engineers and I were proposing as an immediate action was a rewrite of ONLY the cash account module. It would have been possible to write a new, second version of this module alone leaving all of the other code untouched.”
Is that what you’re referring to?
David McDonnell: Yes. So what quickly happened was it was very clear that they took this report, and part of their defence or argument to reject the recommendations that we made was to forget the first part of the sentence where we recommended to rewrite it in part, referring to the cash account, and they focused on “rewrite the EPOSS counter system a whole”, and every conversation that was had after that, and certainly I heard in the testimonies, were conflating the whole proposal to rewrite some of the product into, “It’s too big, it’s far too dangerous, it will introduce more problems”, when in fact, if you understood that it was built out of Lego bricks, you could replace the Lego bricks one at a time starting with the most critical, the most important, which I would argue was the cash account.
Here, you could even – because it was a batch process that wasn’t part of the counter client/customer interaction, you could rewrite that as a separate module and have it running as a shadow process on the counter. You could run the cash account twice at the end of the day or whenever, as a secondary confirmation, and use the replacement module to check the validity of the first one. Once you’d proved that it worked, you could take the old one out and just continue with the new one.
This was not a large task. It was not something that – I couldn’t understand why they didn’t do it, because it was such a – it’s not a small piece of work but relatively small, and you could have done it without introducing any danger to anything else on the counter.
Mr Beer: In terms of what happened then, can we turn, please, to look at FUJ00121099. This is a new document disclosed to us in the last day or so by Fujitsu. It’s not one you will be familiar with, sir.
I think you have seen this today in fact; is that right?
David McDonnell: Yes.
Mr Beer: We can see that it’s a memorandum addressed to you and others from Chris Humphries dated 12 March 1999, entitled EPOSS Product Improvement Options. Having read the report or the memorandum, does this appear to be a response of a type, of a kind, to the EPOSS PinICL Task Force report?
David McDonnell: Yes. So one of the things that came out of the Task Force was, not just the report that we’ve seen today, but there was a further document which I think hasn’t been found that had specific recommendations.
Mr Beer: The recommendations document?
David McDonnell: I believe the items in this memo are lifted from that document. Basically the content is pretty much the same, the recommendations.
Mr Beer: The recommendations document that we don’t have, okay.
David McDonnell: Yes.
Mr Beer: If we just go through it, please, this is a summary of discussions of workshops held on those two dates and includes some post 25 February ‘99, that is, workshop input from Les Ong? Who was he?
David McDonnell: Les Ong was one of the senior testers who was dedicated also dedicated to the Task Force initiative.
Mr Beer: And then you will see Candidate Product Improvements Measures:
“The following measures were identified as possible ways of improving the EPOSS product to enhance its maintainability and to reduce the risk of severe operational problems.”
Then there’s a list of them if we just go down. I think we will find there are 13 of them. So this is a list of problems: “Stock unit dll” – and you will see what the problem is, yes? “Reporting.” Over the page number 3, “Attribute grammar”. Number 4, which I will read out:
“Cash account. Rewrite cash account report as a separate report. Bring it into line with the POCL view of the cash account and align the two reference data models. Possibly do the rewrite in C for performance.”
What does that last line mean?
David McDonnell: A lot of the stuff was written in Visual Basic, which was a lot heavier and slower. C is a language which is much more – it gives a higher performance after it’s been compiled and delivered on to the system.
Mr Beer: 5, “Error Handling”, 6, “Business Rule Validation”, 7, “Logic Threads”, 8, “Comments”, 9, “Tidy Up Code”, 10, “Modularise Code”, 11, “Document [over the page 12] Error Messages”, 13, “Menu Navigation”.
Then the document says:
“Each of the above improvement measures was evaluated against the following set of criteria. The first two criteria tend in favour of implementing a measure and the remaining three tend against.”
So, if we look at the two pluses – I’m going to call them first – benefit:
“The benefit of the implementing a particular improvement measure in terms of the product’s enhanced maintainability (time/effort/risk), stability, and robustness.”
That’s obviously a benefit. Secondly, a plus, the do nothing risk:
“Risk that, if a particular improvement measure is not implemented, a severe software problem will arise in live operation that is difficult or impossible to manage and recover from. This could arise in the initially released system or from an error implementing a subsequent change to the software. There is also the risk that, due to poor maintainability, a business critical change could not be implemented within the required timescale.”
Then, if we go down to what I’m going to call the minuses, i.e. things that tend against doing any of the 13 things:
“Destabalisation Risk:
“The risk that implementing a particular improvement measure will destabilise the product …
“Migration risk:
“The risk that for a particular improvement measure the process of migrating in live service from the old to the new product embodying the measure will encounter unforeseen difficulties, leading to a position that is difficult or impossible to manage and recover from.”
Then over the page, the other negative minus is “The time effort and budget required to implement the measure”.
Then, in a matrix, the author has written “high”, “medium” or “low” against each of those 13. That list on the left-hand side corresponds to the 13 issues we’ve mentioned, and addresses them against each of the five criteria: benefit, do nothing, destabilisation, migration, or cost.
Can we just look at Cash Account. The benefit is said to be low, the do-nothing risk is said to be low, the destabilisation risk is said to be high, the migration risk is said to be high, and the cost is said to be high. Do you agree with those five evaluations?
David McDonnell: No.
Mr Beer: Do you agree with any of them?
David McDonnell: On that highlighted line, when I first saw this document this morning, my initial impression was that I’d got my understanding of positive and negative the wrong way round because they are inverse to what I would have said.
Mr Beer: The benefit would have been high, the do-nothing risk would have been high –
David McDonnell: Yes.
Mr Beer: – the destabilisation, migration and cost risk should be low?
David McDonnell: Yes.
Mr Beer: They should all be the other way round?
David McDonnell: Yes.
Mr Beer: Moving down the page, there’s then a score essentially given to each of the evaluation of high, medium and low and, if you look, that explanation is at the foot of the page. If we go over the page, please, applying those weightings to the scores, if we highlight Cash Account, the benefit has been scored at three, consistent with high; the do-nothing risk 3 also, but then three minus 6s for destabilisation, migration and cost, leading to a grand total of minus 12.
Then, if you go to the table underneath – just scroll down a little bit – the ranking of whether or not to do any of those things is set out in a rank, and rewriting the cash account comes out as bottom. Do you agree with that assessment and approach?
David McDonnell: I don’t understand it. It’s upside down. I’ve no idea how you could ever come up with such a ridiculous scoring system and end up with the cash account being at the bottom by quite a margin, yet you’re talking about a financial accounting system that clearly doesn’t work. It’s beyond me.
Mr Beer: At this stage in the process – we’re dealing at March 1999 – where was the EPOSS project in terms of its ability to get any change done?
David McDonnell: In terms of change, are you referring to bug fixing or additional functionality?
Mr Beer: Both, if you take them in stages, please.
David McDonnell: I wasn’t on the counter team by then, but I believe that they had improved certain things to a degree. They started to put better practices in place and things like that. It was a little bit more disciplined.
I think they had implemented some of the new functionality in a better way than they had historically. But the fact that they are left with a legacy code and all of the associated problems inside it just meant that it wasn’t going to be a different outcome.
Mr Beer: Can you help explain how it was that Chris Humphries, who you have referred to in relatively positive terms in your evidence so far, came to write a document such as this?
David McDonnell: I can’t. I was surprised when I saw it because, exactly as you’ve said, Chris was always very – he understood the problem and he was always very supportive in trying to effect change, to a degree. Why this has happened at that stage in time, I can’t imagine. I have my suspicions, but I should probably not …
Mr Beer: Can you help us with that.
David McDonnell: I think the project by then had – shortly after the cash account – sorry, the Task Force initiative and the subsequent recommendations that came out and the noise around it, there was a very definite project push to get the lid back on to that tin of worms and move on with acceptance in a positive way, and that did not include rewriting any of the code, as evidenced by this recommendation here.
So I think – how – any dissenting voices were either sidelined or moved or ignored, and this was the narrative to move on through the acceptance process. So it doesn’t surprise me that documents like this were created but, you know, as to why, I can only imagine.
Mr Beer: Can I just unpick that a little bit so I understand it. You’re saying that it doesn’t surprise you that a document like this was written, because at this stage of the process the prevailing narrative was to move on, get the project rolled out and, therefore, rewriting the cash account was not on the cards, and you couldn’t write a document that said, “We need to rewrite the cash account”?
David McDonnell: Correct.
Mr Beer: Can we go over the page, please. The second workshop discussion, which I think we saw from the earlier in the report was 9 March 1999, includes the table would be now put into order of doability –
David McDonnell: Mm-hm.
Mr Beer: – with cash accounts still being bottom, and adds some narrative in the right-hand column, and you can quickly look down them:
“Error messages. Desirable but not a measure to enhance the product maintainability and robustness.
“Stock unit dll. Too much effort to do all at once.
“Attribute grammar. Most of the benefit comes from the documentation. Redundant attributes is desirable for space saving, but does not add much towards maintainability …
“Reporting. More beneficial than Stock unit, but also needs prior design before decision can be made as to what to do in this release …
“Document. Should be undertaken as models are open for development work. Waste of time documenting existing stock unit …
“Logic threads. Too risky.
“Comments. Implement as modules are opened up for development.
“Tidy-up code. Removal of obsolete code too risky.
“Modularise code. Not worth it.
“Error handling. Implement high-level trapping immediately and detail error trapping as modules are opened for development work.
“Menu navigation. No.
“Business rule validation. No.
“Cash account. No.”
Does that narrative reflect the view that you have just described?
David McDonnell: Yes, I would say so. It’s – I would disagree with most of the conclusions there.
Mr Beer: You were, I think, an addressee of this memorandum?
David McDonnell: Mm-hm.
Mr Beer: I think that reflects the fact that you were present at one or both of those two workshops?
David McDonnell: I don’t – and I stand to be corrected if I was – I don’t recall being at those workshops. I think I may have been copied, because some of the potential outcomes may have affected the project I was working on at the time, which was APS, but also the fact –
Mr Beer: Sorry to stop you there, to interrupt you. Had you been moved on by now?
David McDonnell: Yes.
Mr Beer: Why were you moved on?
David McDonnell: Wrong-answer syndrome. At the end of the Task Force initiative, Jan and I had written the audit document. Certain recommendations were made. Obviously the cash account rewrite was a major part of that, and then at that point, shortly after that, Terry Austin decided to have a reorganisation of some of the teams, the EPOSS counter team being one of them. As part of that, I was called into his office and asked you – he said, “I’d like you to take over officially and formally from Steve Warwick who will be moving on to the business liaison”, or something similar, “and I want you to be the development manager of the EPOSS counter team.” This, that and the other, “You can have the following resources”, et cetera, et cetera. “Okay, fine.”
I said to him at that point I would accept the position on the condition that we rewrite the cash account, and at that point Terry was frustrated, to say the least. He wasn’t very happy with me putting a condition on that acceptance. It was clear that the cash account wasn’t going to get written. That conversation was very quickly brought to a halt, and I was ushered out of the office, and I never really spoke to Terry after that again. We never really had any interaction.
Very shortly after that meeting, he appointed Phil Hemmingway, who was working in the EPOSS counter team for me as business analyst at the time.
Mr Beer: He was under you?
David McDonnell: Yes, and he appointed Phil Hemmingway as the development lead on that team at that point, and I was moved off on to another team, the LFS counter development team, which was new piece of software they needed developing.
Mr Beer: So you think you probably didn’t attend in February ‘99 and March ‘99?
David McDonnell: I think I was already off on the LFS counter team by then.
Mr Beer: Do you know why you were copied into a memorandum such as this?
David McDonnell: No idea, sorry.
Mr Beer: Did it have anything to do with the work that you were currently doing?
David McDonnell: It’s certainly nothing to do with Logistics Feeder Service, which is where I was working at the time. I can only think that it was as a result of the fact that this document is based largely on the recommendations that Jan and myself and Martin Smith and a couple of the guys made as part of the Task Force; so those recommendations that are in the previous pages are probably lifted from that document, and he might have been copying me in as a courtesy. I don’t know.
Mr Beer: So this didn’t have anything to do with your current work. Can you now recall doing anything as a result of the receipt of it?
David McDonnell: I don’t remember even seeing it, to be honest, at the time. I do recognise the document now that you’ve shown it me, but I have no recollection what happened around it or what result it had.
I think by then everyone had just given up trying to argue with the narrative.
Mr Beer: If we go down to the Conclusion then, please:
“Design activity will be undertaken to design the product improvement implementations for Stock Unit and Reporting.”
I think that’s two of the 13 things:
“It will then be decided what development can be achieved with Release 2+ timescales. This work will be scheduled.”
I think that’s the end of it. Can we move forwards in time, please –
Sir Wyn Williams: Before we do that, Mr Beer, as you rightly point out, I hadn’t seen this document before. Given that, I’m very grateful we have seen it now, but, Mr Whittam, I would like a written explanation as to why it is that this document appears the night before this witness gives evidence, given that we have been debating matters around the EPOSS Task Force for weeks.
Mr Whittam: Certainly. It was in the correspondence that came with it.
Sir Wyn Williams: All right. I will read that and then I will decide whether I want any more. Thanks.
Mr Beer: Can we turn, please, to FUJ00079782. This is an audit report of 28 October 1999. You are neither an author nor an addressee and, therefore, I don’t think you would have seen this at the time. Can you look at just the abstract. You have seen this document before as part of preparation for giving evidence, a report of: “An audit of the CSR+ development activities and presents a snapshot view during September 1999. It details the results of the investigation and provides an opinion as to the state of process compliance and capability.”
Then, if we can move forward, please, to pages 19 and 20, there’s a description of EPOSS.
“From the CSR+ perspective, the development of the EPOSS product has been successful with software drops being made according to the planned schedules and confidence in the team that future drops will … be achieved on time.
“Unfortunately EPOSS continues to be resource hungry in dealing with live problems associated with CSR and in ensuring that these fixes are brought forward and incorporated into the CSR+ product.
“The EPOSS Task Force report –”
And that’s footnote 6 which takes you back to that list of associated documents that I showed you right at the beginning of giving your evidence. It is a reference back to what’s described as version 0.3 of the PinICL Task Force report dated 29 September 1998:
“The EPOSS Task Force report raised the question of the maintainability and resilience of the EPOSS code following the 6 week PinICL blitz where some 550 PinICLs were processed. Since then a further 996 PinICLs have been raised … and these can only have had a detrimental effect on the quality of the code.”
What do you think reading that?
David McDonnell: Sorry?
Mr Beer: What do you think, reading that, seeing that?
David McDonnell: It’s everything we said in the Task Force. I mean, the statistics, the volume of PinICLs alone tell the story. It’s an accurate, a very accurate description of where we were at the time.
Mr Beer: The authors continue:
“In particular, the maintainability, resilience and potential for change aspects must be subject to doubt. The report also identified many instances of poor programming technique and application of coding standards and, while CSR+ changes have been reviewed by the Team Leader no attempts have been made to address the significant body of code not affected. There’s also anecdotal evidence that EPOSS components used by other applications are fragile and cause problems for the calling application, Print Server was mentioned by both LFS and APS counter teams.”
Then if we go over the page, please –
David McDonnell: There’s quite a lot in that paragraph alone that is indicative of the fact that, first of all, the number of PinICLs being raised isn’t diminishing and the problems persist. Sorry, could you just go back up to the page before.
Mr Beer: Absolutely, yes. One page back please, Frankie.
David McDonnell: So they are still suffering from the same problems of having to introduce so many PinICL fixes or additional code, maintainability and resilience is a problem.
Mr Beer: This is remember late October ‘99?
David McDonnell: This is over a year later and they are still suffering from the same issues, poor programming techniques, application coding standards are missing. Okay, so the last part of the paragraph where it’s referring to Print Server. So by then I was off on the LFS counter development team, and my peer on the APS team was also experiencing the same problems that, when we built a separate part of the counter position which had to talk to the EPOSS counter and use the print server, which was part of the EPOSS counter, nothing was working as it should, and that’s evident, that not just the LFS team that I was running but also the APS team upstairs was experiencing the same problem. In fact, that print server became quite a bit of an issue.
Mr Beer: If we scroll down, I maybe jumped too quickly to the next page. There’s an analysis month by month of PinICLs raised, I think, for EPOSS and desktop, and then EPOSS on its own. If you go over the page, please, you, can see that the numbers don’t change substantially?
David McDonnell: So that table in itself tells you exactly what we were referring to earlier, that you would expect to see the number of issues diminish over time as the quality improved and, in fact, there it’s almost becoming worse.
Mr Beer: And then the narrative of the report says:
“The figures indicate that the problems facing EPOSS during the Task Force period have not diminished.”
I think that’s exactly what you said.
David McDonnell: That’s exactly it. So there’s a comment about this CSR+ release, which was a big bone of contention at the time, that at the end of the Task Force they were given the report that we co-authored detailing what the senior engineers, senior auditing guy, and all of the experienced people around the project were saying, detailing the problems.
It’s like the captain of the ship’s been told that there’s a hole in the boat and it’s filling with water by the engineers. Instead of fixing the hole, what they did was they went away and constructed this CSR+ release, which is akin to painting a plimsoll line on the outside of the boat so that they could measure how fast it was sinking.
The whole context of this CSR+ release was about being able to detect discrepancies between the counter and the middle and back office, the APS systems and such, and highlight where there was a difference between the number of transactions or the balance between the two being different.
That’s just building a dipstick instead of actually fixing the hole in the boat. They spent a year, an inordinate amount of time and resource, on this release instead of fixing the problem.
Mr Beer: The authors continue:
“Of greater concern are the non-EPOSS PinICLs within the group suggesting that there are still serious quality problems in this vital, customer-facing element of the system.”
Then in the box in italics:
“The EPOSS solutions report made specific recommendations to consider the redesign and rewrite of EPOSS, in part or in whole, to address the then known shortcomings. In light of the continued evidence of poor product quality these recommendations should be reconsidered.”
You’re shaking your head. Why is that?
David McDonnell: Well, there it is again. This is a year later, and Jan Holmes has even got a box around it trying to emphasise the fact that this decision should be revisited. It’s quite evident here that the quality of the product a year later is no better, if not worse, than they thought it was a year ago.
Mr Beer: Just to remind ourselves, on the distribution list was John Bennett, Mike Coombs, Terry Austin and Martyn Bennett of this report. Can we just look at two other documents, please, briefly. The first POL00043691. Look at page 57, please. This is an AI for 376 which you know about, I think.
David McDonnell: Yes, I’ve seen this document, yes.
Mr Beer: I don’t think you would have been copied in at the time; is that right?
David McDonnell: No.
Mr Beer: Would you have had access to it?
David McDonnell: No.
Mr Beer: In which case I’m not going to ask you questions about it.
David McDonnell: I can just comment on what this paragraph:
“These issues have come to light when comparing a TIP derived cash account with the electronic cash account sent by Pathway.”
So this is a direct result of them measuring the difference between the two areas. So that’s detected a difference between the counter and the back office. So that’s what this is alluding to.
Mr Beer: Maybe then if we look that next page, page 58:
“Pathway has analysed all occurrences of where the TIP derived cash account does not equal the actual cash account. There is no suggestion or indication that there is a fault in the calculation or reporting of the cash account. The incidents relate to an occasional missing transaction when reporting to TIP. This rate of occurrence is around 1 per cent of outlets per week.”
You had obviously been advocating the rewriting of the cash account in autumn ‘98. We’re now here; this was July ‘99. What would your view be as to whether or not the problems referred to in this AI were related to the problems in the EPOSS code, or can’t you tell?
David McDonnell: I think it’s highly likely. I don’t think they know what the problem is, reading that. We saw this stuff all the time, and it’s no surprise to see it there at this time and date. How they’ve based that conclusion that it’s TIP – well, how can the derived cash account … There’s no suggestion or indication that there is a fault in the calculation, but there’s a transaction missing. How does that work?
Mr Beer: Let me turn lastly to FUJ00079783 and turn to page 6 of this document, please. Back a page. Forward a page, please. Thank you.
Now, we don’t as an Inquiry have emails or many emails dating from this time at all, but we do benefit from having the content of an email being cut into, cut and paste it seems, into this other document which we do have, and you will see on the left-hand side a report of an observation or recommendation, being:
“The audit identified that EPOSS continues to be unstable.”
This is now November ‘99.
“… EPOSS continues to be unstable. PinICL evidence illustrated that the number of PinICLs raised since the ‘98 Task Force and the rate of their being raised.
The EPOSS solutions report made specific recommendations to consider the redesign and rewrite of EPOSS, in part or in whole, to address the then known shortcomings. In light of the continued evidence of poor product quality these recommendations should be reconsidered.”
We’ve seen that.
Then the response, 25 November, an email issued by TPA, which I understand to be Terry Austin:
“We have not formally closed down the recommendation that we re-engineer the EPOSS application due to its inherent instability. Since this recommendation was made, a number of events/actions have occurred. We’ve embarked on a major maintenance exercise for LT2 which targeted several known stability issues. In parallel, we carried at a defensive testing activity which identified a number of faults which were addressed. The intensive exercise designed to remove Acceptance Incident 298 resulted in many substantial improvements to the error handling messaging and printing aspects of the product. We finally introduced improved unit and link testing and more disciplined configuration control. Finally, the maintainability and enhanceability of the product has been proven by the speed and quality of the SIP 16 and EPOSS reconciliation developments.
“We will of course continue to monitored the PinICL stack for the next few months and if necessary reevaluate this decision. Would Jan please close this issue formally using the rationale described.”
What do you understand that answer to mean?
David McDonnell: So this is obviously one of the critical elements for POCL accepting the product, and he’s started the explanation there with his first sentence, that they haven’t ruled it out, even though it’s a year later and the PinICL statistics tend to suggest that the product’s worse, not better, yet concludes with a final sentence of asking Jan to close it, based on a logic which is: we’ve done lots of things, remedial work, code improvement, et cetera, et cetera but those improvements are not reflected in the PinICL statistics. So how, if you were the client POCL, or – how could you accept that as an argument, unless you didn’t have visibility of the PinICL statistics, I don’t know.
Mr Beer: Can we just turn then lastly to POCL’s knowledge and turn up paragraph 51 of your witness statement, WITN00620100 at page 17.
I should start actually with paragraph 51 on the previous page. You say:
“I witnessed the liaison between representatives of POCL and ICL Pathway during the design and development of the … system daily, between Steve Warwick and a POCL lady. The discourse and interaction was always very good. POCL had a presence in the building and POCL and ICL Pathway met almost daily. I attended some of those meetings. I understood that the POCL rep, Barbara Longley, was in attendance and Steve Warwick mostly from ICL Pathway.”
Then over the page, please, in the paragraph that you have just given in paragraph 51, were you saying that by those means POCL knew about the serious problems with the EPOSS, and particularly the cash account module within it?
David McDonnell: I think it would be difficult to have been in the building and not be aware of the problems that the EPOS counter system was experiencing. I also think it will be difficult to be in the building without being aware of the number of PinICLs that were being raised, and I know that’s a subjective answer, but I think it will be difficult to argue that you’re in the building but you’re not aware.
Now, having said that, Steve did do a great job at managing the client and fielding all of the problems, and he was a great salesman in that respect. So it could be also equally arguable that they were kept in the dark to a certain degree.
Now, when I refer to the fact that they definitely knew that cash account was broken, I’m talking about the PinICL which I think is referred to that bottom of paragraph 52, where it has a full audit trail of everybody who read, touched and commented on that particular cash account error, and there are at least one POCL person – I think that was Barbara Longley – whose desk that PinICL did cross, and she has commented on it at some point.
So that speaks specifically to that one PinICL. Whether you could argue that that gave them an awareness of all the problems, I don’t know, but I think it would be difficult not to.
Mr Beer: Let’s just quickly then look at that PinICL FUJ00067416. Just before we dive into the detail, is your understanding that the peak incident management system was an internal ICL system?
David McDonnell: I was under the impression – I may be wrong – that the peak incidents came from – I’m not sure if it was the support people but that was POCL-facing. So when POCL raised an issue, it was a peak incident. When that was fed into the ICL PinICL system, that was the PinICL system.
Mr Beer: The call logger on the top right-hand side is shown as EDSC. Was that the European development and support centre at ICL?
David McDonnell: I don’t know what that stands for.
Mr Beer: Let’s look then at the first box in the progress narrative against the date of 16 May, three lines in:
“The host-generated cash account line comparisons report dated 15 May where Post Office 169207 has a difference in the receipt and payments total for Cash Accounting Period 06. Please investigate.”
Would you agree that that suggests that the issue had arisen with Post Office branch 169207?
David McDonnell: Yes.
Mr Beer: So would that be the customer who had raised the issue with the EDSC?
David McDonnell: Yes. In fact, it is logged as a call log or customer call; so that would indicate it’s come from a customer.
Mr Beer: So POCL had an involvement to the extent that a subpostmaster or a person in that branch raised the issue initially?
David McDonnell: Yes.
Mr Beer: I’m more interested in whether POCL design development people, whether co-located or not with ICL, would have sight of or visibility of this document. Can you help us on that?
David McDonnell: Well, they certainly would because it’s – this is out in the POCL estate and it’s them who’s raised the call. I was under the understanding that Barbara Longley, who’s in the second box down there, was a POCL person. I may be wrong; I thought she was. This has come from the POCL estate and it’s gone through right from the customer through first line support.
Mr Beer: So it’s Barbara Longley’s user identification on this peak that causes you to say that POCL knew that there were problems with the EPOSS system?
David McDonnell: Yes.
Mr Beer: Yes, thank you very much. They are the only questions that I ask. There may be questions from others.
Sir Wyn Williams: Anyone? No. I think that’s it, Mr Beer, and your prediction was very precise.
Thank you, Mr McDonnell, for coming to give your oral evidence. Thank you too for making your written statement. I’m very grateful. 2.00 or do you need to start at 5 to?
Mr Beer: That’s all down to Mr Blake.
Mr Blake: 2.00’s fine.
Sir Wyn Williams: 2.00.
(12.53 pm)
(Luncheon Adjournment)
(2.00 pm)
Jan Holmes
JAN ROBERT HOLMES (sworn).
Questioned by Mr Blake
Mr Blake: Thank you very much. Can you give your full name please.
Jan Holmes: Yes, it’s Jan Robert Holmes.
Mr Blake: Mr Holmes, you should have in front of you a hard copy of a witness statement.
Jan Holmes: Yes.
Mr Blake: And is that dated 3 September of this year?
Jan Holmes: Yes, it is.
Mr Blake: Could I ask you to turn to page 18, please. Is that your signature that bottom?
Jan Holmes: Yes, it is.
Mr Blake: Is that statement true to the best of your knowledge and belief?
Jan Holmes: Yes, it is.
Mr Blake: Thank you for coming today. As you are aware, I’m asking questions on behalf of the Inquiry. I’m going to start by asking just about yourself. You began your career in the Civil Service; is that right?
Jan Holmes: Yes, that’s correct. In the mid-’70s I started programming and did some work programming operations management and then moved into computer audit in the early ’80s.
Mr Blake: I think you worked for a few small IT companies for a period?
Jan Holmes: Subsequently, to that, yes.
Mr Blake: Then in 1995 you joined ICL Financial Services Division in Wilmslow; is that right?
Jan Holmes: Yes, that’s correct.
Mr Blake: In 1997 you transferred to ICL Pathway in the role that we’re going to hear about, as audit manager.
Jan Holmes: Well, this is where I have to divert from here a bit. I actually realised that I joined late 1996 in a slightly different capacity, but it didn’t go for very long, and then I was transferred into this role as audit manager in 1970 (sic). So that first bit can be discounted, I think.
Mr Blake: Do you remember what the 1996 post was at all?
Jan Holmes: Yes, it was process engineer, I think.
Mr Blake: Thank you. Then you left that role briefly, the audit manager role, and I think you then came back and returned to the role?
Jan Holmes: Yes, in May 1970 (sic) I left to join a central ICL project calling Propel which collapsed in the middle of 2001 – nothing to do with me but – and then I came back to the programme until about 2008 when I left.
Mr Blake: Sorry, when did you leave?
Jan Holmes: I think it was about May 2000.
Mr Blake: 2000, thank you. You retired in 2008; is that right?
Jan Holmes: No, I didn’t retire. I left the program and then was made redundant the following year, and then did some sort of consultancy work until I finished work about 2015.
Mr Blake: Thank you very much. Just as a matter of transparency, in preparing for this Inquiry, I think you confirm with me you that you had spoken to Terry Austin?
Jan Holmes: Yes. I mean, I have to. I’ve known Terry since we started in the Civil Service because we were in it at the same time. So I know him as a friend as well as a work colleague and, in fact, it was Terry who steered me into Pathway in ‘96/97.
Mr Blake: I think you may have spoken to one or two other people; is that right?
Jan Holmes: Yes, I did, but nobody who was involved with this.
Mr Blake: Thank you. When you began your role as audit manager, you have described the audit function as being immature. What do you mean by that?
Jan Holmes: To all intents and purposes, it was a standing start. What happened was, when I came in as process engineer, I was doing that kind of work, and then it became clear that, because of the volume of audit information and data that was going to be collected, and what was expected of it through the requirements, that there was a need to take a more active management role in it.
Also the fact that, because I had been through the Institute of Internal Auditors training, along with a whole raft of other civil servants – it was the result of a National Audit Office initiative – it meant that there was consistency in approach between what I would be doing and what the Post Office and Benefits Agency will be doing as well, the idea being that we could work a bit more harmoniously because we had the same expectations of standards and practice and what have you.
In terms of the audit function with regard to the data and the trails, yes, that was a standing start because they were just being developed and, in terms of the internal audit side, that was a new function that I was asked to put in place to interface and work with Benefits Agency and the Post Office.
Mr Blake: Thank you. The broader team that your role fell into, that was the Quality and Risk Management Group; is that right?
Jan Holmes: Yes, that’s correct, under Martyn Bennett, who was on the senior management team.
Mr Blake: Do you remember who else was on the senior management team?
Jan Holmes: The senior management team?
Mr Blake: Yes.
Jan Holmes: John Bennett ran it, Liam Foley was the Business Development Director, Tony Oppenheim the Financial Director, Terry Austin, I think at the time was on the board as the Program Director. I think there was an HR guy as well, John Hobson.
Mr Blake: Were you aware of how it worked, that senior management team? Did it have regular meetings for example?
Jan Holmes: I believe so, yes.
Mr Blake: Did you attend any of those meetings?
Jan Holmes: No.
Mr Blake: Do you feel that you had enough support in your position from the board or from the senior management?
Jan Holmes: Yes, I do. It’s not often when you set up an audit department that you get the level of support that you want. You often have to fight your corner to get heard and listened to and taken seriously. But I have to say that on this account, this project, it was pretty good. Visibility was good to the senior management team through reports and corrective actions. Martyn was quite vociferous in supporting the role in his position as my boss.
So I was quite comfortable with it compared to some I’d been in where, you know, internal audit was the unfortunate noise in the corner.
Mr Blake: At paragraph 14 of your witness statement, you have listed three audits that you consider that are most important to the Inquiry. There’s no need to go to it in the statement unless you would like to.
Jan Holmes: Yes.
Mr Blake: You describe the CSR+ development audit, the EPOSS Task Force report, and the implementation audit. I’m going to take them in chronological order, just to remind ourselves of what each one of those are, although we have heard quite a bit already this morning.
Can we look at a FUJ00080690. That this EPOSS PinICL Task Force report that we have seen a lot about this morning, and it talks about the Task Force carrying out its work in the summer and autumn of 1998.
Jan Holmes: Yes.
Mr Blake: I’m going to ask you shortly about the various dates of the reports, but perhaps we can just turn to page 7 for this purpose just to remind us of what it says. There’s a key section there about the EPOSS code, and it highlights code decay through fixes, for example, PinICLs likely to increase, and poor workmanship. Do you agree with that? Is that a fair summary of what it says there?
Jan Holmes: Yes.
Mr Blake: Was that something you agreed with at the time? I mean, you wrote the report. Presumably that was your view.
Jan Holmes: Yes. I mean, just to put these things in context, though just for a minute, audit reports are by virtue of their nature snapshots in time, okay? So, when you go in to do an audit, you go in and look at the evidence as it is over that one-or-two-week period that you’re doing the work. You then take that away, you sit and look at it and think about it, have conversations with other people, and then you draw conclusions, and you ask yourself: if this were to continue, what could go wrong? That’s when you start putting forward the scenarios of code decay and problems coming up in the future.
You are trying to contextualise what you’ve found into what potentially could happen if this was allowed to go on. So, yes, code decay could easily become an issue, but I’m not in a position to say in 1998 what that issue might be in the long-term.
Mr Blake: But you’re looking forward there, predicting likely future problems?
Jan Holmes: Yes, in that context I am, yes.
Mr Blake: Then we’ll look at the CSR+ audit. That’s FUJ00079782, please. I should say that the first one I think was distributed to a similar distribution list, Martyn Bennett being your manager?
Jan Holmes: Yes.
Mr Blake: I think he received all of these reports, but we can have a look at that. This was the summer and autumn ‘99. We’ve looked at this already this morning, but let’s look at page 20, please. This paragraph we looked at just shortly before lunch says:
“The figures indicate that the problems facing EPOSS during the Task Force period have not diminished. Of greater concern are the non-EPOSS PinICLs within the group suggesting that there are still serious quality problems in this vital, customer facing element of the system.”
So I think you have just said that, in relation to the report of the Task Force, you were making some predictions or looking to likely future events.
Jan Holmes: Yes.
Mr Blake: This is somewhat later, and it looks as though those events have been realised.
Jan Holmes: Correct, and that’s supported by the evidence in the table that sits above it.
Mr Blake: Then it goes on and says:
“The EPOSS Solutions Report made specific recommendations to consider the redesign and rewrite of EPOSS, in part or in whole, to address the then known shortcomings. In light of the continued evidence of poor product quality, these recommendations should be reconsidered.”
Can you tell us: do you remember what the EPOSS solutions report was?
Jan Holmes: No, unfortunately this is a lost document. I can’t find it in my own personal archive, and it looks like you haven’t been able to find it either. But it was written as a follow-up to the Task Force report. I’m almost certain Dave McDonnell wrote it as a technical report, opposed to an audit report but I just don’t know what happened to it, where it’s gone.
Mr Blake: We have a note of it being dated 21 September 1999.
Jan Holmes: Mm-hm.
Mr Blake: Does that fit with your recollection?
Jan Holmes: What, of the solutions report?
Mr Blake: Yes.
Jan Holmes: Quite possibly.
Mr Blake: Do you remember in broad terms what it may have said?
Jan Holmes: No. I think it may well have just gone into more technical detail about what we found in the EPOSS report, what we reported in a sort of more general managerial-type sense, as opposed to a deep technical sense.
You have got to bear in mind I’m not a technician. So the bits in that report and the EPOSS Task Force report that were technical, I had to get contributions from people like Dave.
Mr Blake: We’ve seen in the EPOSS PinICL Task Force reports some pretty frank language there. For example:
“Whoever wrote this code clearly has no understanding of elementary mathematics or the most basic rules of programming.”
Jan Holmes: Yes.
Mr Blake: Is the EPOSS solutions report likely to have addressed that at a more technical level?
Jan Holmes: Quite possibly, yes.
Mr Blake: So would its finding be consistent with EPOSS –
Jan Holmes: Yes.
Mr Blake: Then the third document that you’ve referred to as the implementation audit, and that at FUJ00079788, please, and that’s around the same time, so we’re looking at summer/autumn 1999. Again, if we scroll down, this went to Martyn Bennett. It went to yourself. You didn’t actually write this report; is that right?
Jan Holmes: No, that’s right. Stanley Loam did.
Mr Blake: Then if we look at page 5 there’s the Management Summary. Thank you very much. I don’t think we need to read the Management Summary in depth, but would it be fair to say that there was some criticisms there focusing on lack of processes and procedures and documentation in certain areas?
Jan Holmes: Yes, I think so, yes.
Mr Blake: Thank you very much. That can be taken down.
All of those three reports that you have mentioned as most significant for this Inquiry, am I right in saying that they all went to the senior management of ICL Pathway?
Jan Holmes: With the exception of the EPOSS Task Force, no, that had a very restricted distribution, and the reason for that being Terry commissioned that piece of work because he wanted to know what was going on. So, therefore, when I did the work through the Task Force, the report went to Terry and it was up to him what he did with it. It wasn’t – to my mind it wasn’t a general distribution document.
Mr Blake: But it did go to Mr Bennett as well?
Jan Holmes: Well, as my boss, yes.
Mr Blake: Were you aware of him raising it with senior management, the people that we’ve discussed?
Jan Holmes: No, no, not aware of that.
Mr Blake: Do you think it was likely or unlikely that it would be discussed?
Jan Holmes: I really don’t know. I can’t comment.
Mr Blake: What about sharing with the Post Office? Are you aware of any of those three documents or their substance being shared with the Post Office?
Jan Holmes: No. It would be highly unlikely for audit reports to be shared with the Post Office. I think, in keeping with most audit departments, the reports are for management use purpose. If they choose to share them, that’s up to them. But, as an auditor, I’m not at liberty to just distribute reports willy-nilly.
Mr Blake: To the best of your recollection, in relation to those three reports, were they shared?
Jan Holmes: Not to my knowledge.
Sir Wyn Williams: Before Mr Blake goes on, I’m sorry, I missed a detail. You did tell us who you thought might have authored the implementation report and I missed it.
Jan Holmes: Sorry, his name was Stanley Loam, L-O-A-M.
Sir Wyn Williams: Thank you.
Jan Holmes: He was employed as an auditor at the time.
Sir Wyn Williams: Thank you very much. Sorry, Mr Blake.
Mr Blake: Not at all.
Would he have fallen underneath your level? Were you his manager, or were you on the same level; do you remember?
Jan Holmes: No, I mean – yes, organisationally he sat below me on the chart but, you know, we worked as equals.
Mr Blake: I’m going to move on and concentrate on that PinICL Task Force because I just want to see if you can help us at with this document, this date issue that we’ve been looking at this morning. It may be that you are not, but let me take you to the documents to see where we get to.
Let’s look at FUJ00080690. This is the main one that we have been working from, and you will see there it’s called version 1.0 and it’s dated 2001. But we know, of course, that the Task Force itself took place in 1998.
Can we now look at FUJ00121098. This is a version that Mr Beer took Mr McDonnell to. That is referred to as in the top right-hand corner version 0.1 with a date of 16 February of 2000.
Then, if we go to FUJ00079782 and can we go to page 2 of this, this is the CSR+ development audit. If you look at page 2 and we scroll down page 2, it refers to associated documents there. Could we scroll down and if you look at item number 6, it has there, “IA/REP/008 version 0.3” dating back to 29 September 1998 Report on EPOSS PinICL Task Force.
It might be worth noting just really for the record, at paragraph number 7 below, there is a report there to the document that we can’t find, which is the Report on EPOSS solutions.
Are you able to help us at all about these various dates and versions?
Jan Holmes: Okay, well, version 0.1 that’s dated September ‘98 – yes?
Mr Blake: ‘98?
Jan Holmes: Yes, the first version 0.1 ‘98, was the report I wrote in the first place, okay? Version 1, if you notice in the Document History section, it says “Move to 1 as an admin exercise”. That was just to move it up to version 1 to get it out of draft. That was all. That was me; I did that. I don’t think – there weren’t any changes to it at that time. It was just me moving it forward.
0.3, I can only assume that some work had been done on the document up to 0.3 later in September ‘98 and that would have been sat in PVCS which was the document repository. So you got 0.1.
Mr Blake: 0.1 seems to be dated 2000 rather than 1998.
Jan Holmes: Can you show me.
Mr Blake: Yes, that is FUJ00121098. If it helps, it was just the document we saw just before. So in the top right-hand corner there –
Jan Holmes: I don’t – I sort of don’t recognise that as a date that corresponds to the version.
Mr Blake: If we go back to FUJ00080690 – and that’s the first one that I took you to and it’s the one that we’ve been working on – you’ve had time to consider that document. Do you consider that that is the original version that you authored?
Jan Holmes: Yes, yes.
Mr Blake: So you’re not aware of any substantive changes at all?
Jan Holmes: Well, no, not unless I made them, and I wouldn’t know what they are now. But there would be nobody else who would have updated that document.
Mr Blake: Thank you.
I’m going to move on to 2000. Can we look at WITN04600104, please. In your statement it’s page 6, paragraph 9f. We don’t need to go to it, but you say you had concerns that continual change of EPOSS code would generate instability, and that’s something we’ve heard this morning as well.
Jan Holmes: Yes.
Mr Blake: Having written the original report, the EPOSS Task Force report, like Mr McDonnell, were you particularly concerned about the EPOSS product?
Jan Holmes: Probably not in the same way that he was. I was concerned from the point of view of its impact and effect on the remainder of the programme, whereas he was I think particularly interested in EPOSS as the product, because he was that person at that time. My concern was just more of a general – the state of the programme moving forward, if it’s got unstable or products that don’t work sat in it.
Mr Blake: Did you have a significant concern about that?
Jan Holmes: Well, define “significant”. I mean, it’s …
Mr Blake: On your list of concerns, how high up –
Jan Holmes: How high up the scale is significant? Yes, I would say it was significant, yes.
Mr Blake: This document schedules the response to certain corrective actions.
Jan Holmes: Yes.
Mr Blake: Can we look at page 9, please. We have seen this this morning, so I can probably take it quite quickly.
Jan Holmes: Yes, I noticed this morning – sorry, if I can just say, when you were talking to Dave, you stopped at the bottom of page 9. There is more to this corrective action overleaf.
Mr Blake: Yes. There are actually two versions of this document. One is an earlier version. This is a later version. We will scroll on in a minute, and it will have the full detail, and I will definitely take you to that.
Let’s start at this page, though. On the left-hand side it says that:
“The audit identified that EPOSS continues to be unstable. PinICL evidence illustrated the numbers of PinICLs raised since the 1998 Task Force and the rate they are being raised.”
We are talking here May 2000, although this document is a document that keeps on getting updated, isn’t it? So it dates back at least before November 1999, because that’s November at the top. 17 November is the first entry on this particular page.
Jan Holmes: Yes, in fact, 17 November is the first entry for all of the actions, which is where I had the original discussions with Mike Coombs, to agree or disagree with the proposed action or the proposed recommendation, sorry, and therefore what was to be done about it. That’s why all the initial entries for this are dated 17 November.
Mr Blake: Thank you very much. On that left-hand side, it refers again to the EPOSS solutions report.
Jan Holmes: Yes.
Mr Blake: So presumably you would have seen EPOSS solutions report at that time?
Jan Holmes: At the time, yes, I guess so.
Mr Blake: It refers to the poor product quality, and recommendation that it should be reconsidered, and this is the follow-up.
If we look at 25 November 1999, there’s the email that we heard about this morning from Terry Austin, about not formally closing it down.
Jan Holmes: Mm-hm.
Mr Blake: Is it right to say that he asks you essentially, “Can we close this down”?
Jan Holmes: Yes, he did.
Mr Blake: Why would he be asking you?
Jan Holmes: Because I suspect, in his opinion, he felt that the explanation he provided was sufficient to warrant closure. It becomes a bit of a judgment call. If you don’t have any objective closure criteria, then you can’t say, “Yeah, you’ve done that, so close it off.” It’s more an agreement between people, groups.
Mr Blake: Yes, because this was a recommendation that it should be reconsidered rather than it has to be done and, therefore, if it had been a recommendation that it needed to be done, presumably you couldn’t close it until it was done.
Jan Holmes: Well, I mean, to be fair, a recommendation is always only ever that, isn’t it, a recommendation? You can recommend that it is rewritten, full stop. If that’s not taken by the organisation, there’s not a lot you can do about it.
Mr Blake: Then shall we scroll down, because then we see what happens after that. 8 December, JH – we’ve heard that’s you; do you agree?
Jan Holmes: Yes.
Mr Blake: Yes. So:
“Jan Holmes requested statistics on fixes delivered to live from RM.”
Now, there was a suggestion that was release management; is that right?
Jan Holmes: Yes.
Mr Blake: Thank you.
“Also informed TPA [Terry Austin] that requires agreement of MJBC [that’s Mr Coombs] –
Jan Holmes: Mike Coombs, yes.
Mr Blake: – “before this can be closed.”
Can you tell us about that entry.
Jan Holmes: Yes, I wasn’t prepared to accept Terry’s request on the strength of what he’d said. So I wanted to get some further information about PinICL numbers and fixes, and I wanted to involve Mike in the closure as he was the Programme Director. Still Terry at that time was just the Systems Director.
Mr Blake: Is the background to those reports that we’ve seen that were quite critical, and are we to infer from that that you saw it as something that needed quite a great deal of scrutiny?
Jan Holmes: Yes, yes.
Mr Blake: The next entry:
“Mr Coombs confirmed that, unless RM statistics contradicted reports provided by PJ [that’s Mr Jeram?]
Jan Holmes: Pete Jeram.
Mr Blake: – “the recommendation could be closed and then the 7th” –
Well, let’s stop there. What was going on there?
Jan Holmes: Well, the way that one’s written, I was waiting for that information to come forward to move the corrective action along, but clearly nothing was happening. This information was not forthcoming from wherever, release management or from Pete. So that’s why in April I tried to force the issue by providing the information that I was waiting for. I know that sounds a bit peculiar, but sometimes you have to do it that way in order to elicit a response.
Mr Blake: So we have your entry for 7 April and that’s “Email …” Presumably from you then –
Jan Holmes: Yes.
Mr Blake: “… to Mr Coombs, Mr Austin and Mr Jeram providing details of release management EPOSS fixes to live. Asked for confirmation that matched PJ [that’s Mr Jeram] reports. If does then will close.”
What were Mr Jeram’s reports that are referred to there?
Jan Holmes: I think as the – Pete was a release manager, I think, at that time. So he was, in his role as release manager, obtaining reports of the status of errors and PinICLs and what have you. Now, it’s a bit moot, because I think the source of both of those pieces of information was the same report. So the release management report was in fact the same report that was going to Pete, so that he knew what was going on. I don’t know that for sure, but I think that was the case.
Mr Blake: So it seems as though you’ve obtained details directly from release management –
Jan Holmes: Yes.
Mr Blake: – which are likely to be exactly the same as Mr Jeram’s figures, or not?
Jan Holmes: Quite possibly, but the issue here was trying to get the thing closed, because it could just have stayed open for, you know, months and months and months more.
Mr Blake: And how were you able to obtain that information?
Jan Holmes: Probably asked for them.
Mr Blake: On, say, 8 December 1999 presumably Mr Jeram could also have asked for them?
Jan Holmes: I believe so, yes.
Mr Blake: Would he have had access to them as of right? Would he have had to ask somebody? Would they be emailed to him?
Jan Holmes: I don’t know whether he would have to request them or whether he got them.
Mr Blake: I think we’re going to come to it, but I’m interested in why it may have taken so long to respond to you on this occasion. Let’s look at the May entry, 3 May:
“Reminder email sent to above, seeking early response. Chased on same day.”
So again is that you chasing a response?
Jan Holmes: Yes. I mean, it’s perhaps a little indicative of the fact that sometimes as an auditor you do have to chase management to actually do their end of the deal. So that was a bit disappointing that it took from December through to May and involved reminders to get things done.
Mr Blake: Then I’ll just read that final entry. So this is the response from Mr Coombs. We’ve heard it before, but just for the purpose of the record and to refresh your memory, it says:
“As discussed, this should be closed. Effectively as a management team we have accepted the ongoing cost of maintenance rather than the cost of a rewrite.”
The reference to management team, is that the same as the management team that we discussed at the beginning of your evidence or –
Jan Holmes: I believe so I, yes.
Mr Blake: “Rewrite of the product will only be considered if we need to reopen the code to introduce signature changes in functionality. We will continue to monitor the code quality (based on product defects) as we progress through the final passes of testing and the introduction of the modified CI4 code set into live usage in the network. PJ [Mr Jeram] can we make sure that this is specifically covered in our reviews of the B&TC test cycles.”
Did management ever go back to you after that occasion to say that they had made a mistake or that they were still reviewing?
Jan Holmes: No, no. Once it’s closed, it’s closed.
Mr Blake: Now, I’m going to just take you to a couple of documents. I know you’ve been following this Inquiry, so you will have seen this documents put to other witnesses. I’ll just briefly look at them.
Can we look at FUJ00058190. Did you receive these month reports?
Jan Holmes: No.
Mr Blake: Can we just look at page 24, please, and it’s the part under Acceptance Loose Ends that I took another witness to, and it’s the second paragraph there. Sorry, can we just go back to the first page.
So we’re in February 2000 here. So this is before that final response and closure. Page 24, please. Thank you. Just below Acceptance Loose Ends, it’s that second bullet point:
“We have dealt with queries from POCL concerning AI376. One formal letter has been responded to, attempting to avoid the conclusion that we have not found EPOSS reconciliation incidents that we should have found, or that we have not reported those we did find. In reality CS are greatly hampered in ‘spotting the incident’ because the reports have not had fixes implemented and report large amounts of do-nothing information. We have attended the Release Management Forum and proposed some reordering of the fix backlog, but it will be at least until the first week of March before this situation improves.”
I’m just going to take you to one other document that has been put to other witnesses and that’s FUJ00079332. This is an email we think it’s 1 April 2000 although it could be 4 January depending on whether it’s an American or British format.
I’m just going to read the first paragraph there. It says:
“We are getting an increasing number of PinICLs on the end-to-end system handling of the new CI4 transaction modes … leading to cash account mis-balances and reconciliation errors. These PinICLs are generally being batted about between different areas.
“I suggest a workshop is set up, led by either Requirements or EPOSS, to present the current end-to-end solution, identify the problem areas and then agree the necessary changes to achieve a consistent solution”, et cetera.
So those are just a couple of examples I have taken, February 2000, April 2000. We know from that schedule of corrective action that you had chased in December ‘99, again in April 2000. Are you aware of anything going on behind the scenes in relation to problems with the EPOSS product?
Jan Holmes: No.
Mr Blake: Looking at these examples, do you think that there was a problem going on behind the scenes?
Jan Holmes: Well, looking at the examples, yes, I think there probably was.
Mr Blake: You say at paragraph 9i of your witness statement, given the senior management involvement with the resolution, you considered this to have been adequately resolved at the time, this being – essentially I think it’s the email from Mr Coombs.
Jan Holmes: Yes. In the context of the corrective action in the audit, yes, I did.
Mr Blake: You phrase that in past tense. You considered this to have been adequately resolved at the time. You, I know, have followed this Inquiry very carefully. Do you now consider that it had been adequately resolved at the time?
Jan Holmes: Well, it would appear not, wouldn’t it?
Mr Blake: Another thing that you say in your witness statement is you refer to a “test error-identify bug-code fix-re test cycle”. I have no idea what that meant; if you can clarify that.
Jan Holmes: Yes. The question was asked in the lead-up to the witness statement about PinICL fixing culture, which is a phrase I used in the EPOSS Task Force report, and in fact it was a phrase that David used this morning.
Really, it was just a case of: “Bug, what is it? Fix it, put it over the wall, test it, hand it away.” It was in the context of, I didn’t see any evidence or any efforts being made to actually stand or sit back, stand back from the problem and say, “Do we have any root causes that we can attribute these problems to and, if we do, what are they?” I didn’t see any of that happening and I wasn’t made aware of any of it happening. It was just a case of, “Oh, here’s another bug. Let’s fix it and let’s move on.”
Mr Blake: Is that something you considered at the time or is that on reflection?
Jan Holmes: No. I considered it at the time. You know, I made the point in the audit report that that was going on. It was just, you know, fix and move, fix and move, fix and move and, you know, we all know that that’s not necessarily the right way to do anything.
Mr Blake: I’m going to move on to a totally different subject. Are you okay to go on?
Jan Holmes: Yes.
Mr Blake: Thank you. That’s audit trails, and it’s again something that’s been of interest to, in particular, some of the Subpostmaster Core Participants. Can we look at FUJ00118196, please.
Now, the date we have at the top of here is 10 November 1999. I can say we have later versions of the same report. They are materially the same as this one in respect of the point I am going to ask you about, but do say if you disagree.
Jan Holmes: No, I agree. The later versions were synced in with later releases of the product where the content of the audit trail may have changed. That was all it was.
Mr Blake: This one is around the time of the roll-out?
Jan Holmes: That’s correct, yes.
Mr Blake: Can we look at page 16, please. I won’t read that out, but it sets out audit trail retention periods, and it says that the normal retention period would be a period consistent with the Companies Act or 18 months, whichever is longer, and that in the case of prosecution support this may be longer in accordance with requirement 829. Is that right?
Jan Holmes: Yes.
Mr Blake: Can you tell us about requirement 829?
Jan Holmes: Yes. Requirement 829 was to do with the efficacy of the audit trail in relation to prosecutions and being evidentially submissible in court in line with PACE. I’m no expert on PACE and in fact I’m not trying to shirk it here, but this requirement was actually a security requirement and not an audit requirement. So, in terms of making it happen, if you like, at acceptance or whenever, we weren’t involved.
What we did do in audit, though, was we made every effort during the design and building of the audit solution with the operational audit trail in it, the integrity of those records was maintained from the point of generation to the point of extraction onto a CD-Rom to be given to the Post Office to do with whatever they wanted.
Mr Blake: Thank you very much. I’m going to very briefly take you to the codified agreement. It is a very lengthy document. We’re only going to look at a short passage which I am sure you have already seen having followed the Inquiry. It’s FUJ000000071. Thank you very much. That’s the codified agreement that was signed in the summer of 1999. Can we look at page 97, please.
It’s 4.1.8 and 4.1., 9 which address prosecution support, and again it refers there to the requirement 829, so this part of the contract being part of that requirement or addressing that requirement.
Is this something that you were well aware of at the time?
Jan Holmes: Yes, in the way it – sorry, the way it impacted on the operational audit trail, yes.
Mr Blake: We will return to this shortly but, if you weren’t told within the 18-month period that there was an investigation and that certain data would be needed for that investigation, would that information not be retained?
Jan Holmes: Correct.
Mr Blake: I think we’re going to look at Cleveleys incident shortly. Was there a subsequent change of position in relation to that retention of data?
Jan Holmes: I think that was probably when Network Banking came in and there was an obligation to retain data for seven years, in line with the commercial audit trail records. I think it’s probably worth going back over the audit trail functional statement, just to explain the difference that existed. Whether it’s valid or not remains to be seen now.
The audit trail functional spec was the stepping stone between the requirements and the system. It identified two types of audit trail: the commercial audit trail, which we hooked into the term records in the contract, and the operational audit trail which was the record of transactions and events that occurred through the system, okay?
We had different rules of access for each of those trails. The audit of the access to the commercial audit trail records would usually be done through physical audits of offices, and records, and filing cabinets, and what have you. The operational audit trail was all held electronically and stored on magnetic tape for the required period, and then had to be extracted back out and put onto CDs for delivery.
Mr Blake: I want to go over a dispute that occurred in early 2000 about the retention of information. I’m going to start in the summer of 1999. Can we look at WITN05970134, please. Thank you very much. This is a document that Mr Folkes was taken to. Did you see his evidence?
Jan Holmes: No.
Mr Blake: Mr Folkes told us that, I think, he produced the words in the boxes. So it’s a response to the words written above, and it’s a Post Office document. You wouldn’t have seen this at the time?
Jan Holmes: No.
Mr Blake: We know from the top paragraph that it at least occurred after 15 July 1999, because that’s when the terms of reference were agreed. Can we look at the first two substantive paragraphs under POSIS Investigations at Outlets, please. I’m going to read this, for the record. It says:
“We were extremely concerned to be informed during the review that POSIS …”
Who were POSIS?
Jan Holmes: No idea.
Mr Blake: “… currently don’t have access to archived data from the system. Data on the [live] system is compressed and archived after 35 days. It was originally intended that access would be gained via the Fraud Risk Management Server, which formed part of the benefits payment system and has now been withdrawn. This means the business could be in a position where it is unable to investigate potential frauds or prosecute cases due to the unavailability of critical data.
“Bob Martin, External Crime Manager for POCL’s Security Investigations Executive, informed us that this issue has been raised with senior managers from the product. Les Thorpe, Investigation manager in the North-East Region, advised us that Pathway had estimated the cost to reintroduce the Fraud Risk Management Server to be in the region of £180,000 with an additional fee of £1,500 per manday for performing extraction. These concerns were highlighted after a possible fraud at Grange Park SPSO which is involved in the Horizon live Trial.”
Do you remember anything about the Grange Park case?
Jan Holmes: No, no.
Mr Blake: Were you involved in discussions at this time with the Post Office about the retention or data and the cost it might involve, for example?
Jan Holmes: Not with retention, because that was already sorted out through existing solution. We did have, or I did have a number of conversations – this was before BA dropped out – with both BA and Post Office internal audit, about what their retrieval requirements were for bulk data at the data centre, so the operational audit trail. They really didn’t know what they wanted, how many they wanted, how they wanted it presented, how they were going to ask for it.
I suppose it was fairly obvious that, from a Post Office point of view, one of the key factors would be: what’s the Post Office number, because we need to get information over a period of time for that post office, and for BA it would be: what’s the National Insurance number, because we want to get information about that over a period of time.
But there was very little in terms of detail, and it took quite a lot of time to get it out of them. Now, the question of the numbers or – well, that’s that piece.
Let’s look at FRMS. As you are probably aware, FRMS was a BA service, and that was what they were going to use to do their investigation. But you have to realise it would have been populated with BA-specific data. There would have been no Post Office data on it. So, as it stood, it would have been no use to the Post Office as a fishing tool.
So they were left then with a retrieval service which was based on minimal requirements. I think we reckoned we’d be able to do about one a month because it was quite intricate, the way it had to be done. So we felt with existing staffing we could do perhaps one or two a month, but that was it.
Sorry, that was also assuming we were dealing with Post Office internal audit, not with various Post Office investigation units who numbered hundreds of people around the country suddenly saying, “We want this data, we want this data, we want this data.”
Mr Blake: We’ll come to that particular disagreement. If we could scroll down slightly, we have the box there from Mr Folkes, and in the second paragraph he says:
“… the replicated copy of [the] data is held at the data centre and is retained for 90 days within the Riposte messaging store. Every record is written to tape to be retained for the period for which we have contracted, (usually 18 months, but extended indefinitely if specific records are needed for the support of a prosecution) and to be made available to POCL under the contracted audit requirements.”
So it highlights that, I think, POCL has access to the data centre; is that right?
Jan Holmes: Yes. Not direct access to the data centre; that was through the retrieval process.
Mr Blake: When you say the retrieval process?
Jan Holmes: The retrieval process that was run by Pathway.
Mr Blake: Is that the request for information process?
Jan Holmes: Yes.
Mr Blake: We will come to that. If we can go over the page, he says POCL’s TIP system had transaction data that could also be obtained by POCL itself. Would you agree with that?
Jan Holmes: Yes.
Mr Blake: Thank you. Then there’s a section on Fraud Risk Management Service and in bold letters there it says:
“The BA Fraud Management Service was never intended to act as a means of accessing specific POCL transaction data, and indeed being a BA service the non-BPS transaction data should not have been able to BA (for sound commercial and data protection reasons).”
Is that what you were just saying to me?
Jan Holmes: Yes.
Mr Blake: So that particular management service was built for the Benefits Agency, not for POCL?
Jan Holmes: That’s correct, yes.
Mr Blake: If we look at number 6 there, it says:
“If POCL did wish to establish its own FRMS, geared to POCL’s data (and, therefore, not restricted to only BPS-related activities), this could of course be done, and Pathway would undoubtedly be willing to provide this service. However, this would be a new requirement, and would need to be funded. Note the distinction here between a full FRMS, with sophisticated data mining and reporting tools, aimed at detecting patterns, et cetera, and simple access to transaction data, which FRMS was never intended, to provide.”
So he seems to be saying there that POCL could have a similar service but it would need to be paid for. Were you ever party to those kinds of discussions?
Jan Holmes: No.
Mr Blake: I mean, he identifies it might involve reporting tools aimed at identifying different patterns, for example, rather than simple access to transaction data. Is that something that you ever considered?
Jan Holmes: No. If that was part of the activity that the FRMS could supply, then I wasn’t aware of that. That was run by a different part of Martyn Bennett’s organisation. But yes, I mean, all we were providing through the audit data was simple transaction data. We weren’t doing any analysis of it, we weren’t doing any manipulation of it. We were just pulling the records off the archive, putting them onto CDs and sending them on.
Mr Blake: So you wouldn’t, for example, look at trends that are relevant to the prosecution of postmasters?
Jan Holmes: No, I mean, why would we? We’re the suppliers of the service. We’re not involved in, you know, running the Post Office’s business other than providing data that they might need.
Mr Blake: Can we scroll down to see 7 and 8, please. Audit Access to Transaction Data. It says:
“A documented process has been established between POCL and ICL Pathway for access to historical audit information from Pathway’s central system, under the auspices of a requirement on Audit (requirements 699 and 829), using a ‘request for information’ procedure.”
So again that’s something I will take you to shortly, but you had a formal procedure to access audit data; is that right?
Jan Holmes: Yes, that’s correct, because we had had a couple of instances where we’d been getting requests in from various people within Post Office, and we probably tried to deal with them with best endeavours. But then, you know, it soon became onerous and we, therefore, had to work out a way of doing this properly. That’s when Hilary and I agreed, and then Chris Paynter that we would have single point of contact, and this was to filter down all of these various investigation units that suddenly seemed to say, “Well, we want dibs on this data.” So …
Mr Blake: Did that all come as a surprise to you?
Jan Holmes: Yes.
Mr Blake: Was it something that was discussed during the earlier stages?
Jan Holmes: No, no, not to my knowledge.
Mr Blake: Looking at paragraph 8 there, this again refers to Grange Park case where the request didn’t follow the correct route, so presumably not through a request for information. It talks about a potential opportunity to improve the agreements between yourselves.
Jan Holmes: Yes.
Mr Blake: Does that jog your memory at all about Grange Park?
Jan Holmes: No, not as an individual Post Office, no.
Mr Blake: Do you remember cases in the early days when there may have been a refusal for audit data because it didn’t follow the correct procedure?
Jan Holmes: No. I mean, I know I was involved in seven or eight enquiries, a few of which I supplied witness statements to, and possibly some I also supplied data to, because at that time it was me that was doing the retrieval work. So yes, it’s possible I was doing that. But then that was on the best-endeavours basis, and it was only when it started to get out of hand that we had to put in place a gatekeeper role which was Chris Paynter.
Mr Blake: Can we go over the page, please, to the paragraph that begins “Bob Martin”. It says there:
“Bob Martin also advised us” – this is from the original Post Office drafting, not the addition –
“… also advised us that Security and Investigation Executive had requested an expert witness statement from Pathway to support a prosecution and this had been refused on grounds that there was no contractual requirement. John Cook advised us that there is a contractual requirement for Pathway to ensure the system meets the requirements of the Police and Criminal Evidence Act. There is a need for Pathway to agree with S&IE and Internal Audit how this requirement will be met, as well as the procedures for obtaining this evidence when needed for prosecutions.”
Do you remember the example that’s given there about a request for an expert witness statement that had been refused?
Jan Holmes: No, no. As I say, I produced several, initiated by different people. I produced one initiated by a police sergeant as part of his investigation. So to some extent at that time, as I said, it was best endeavours, and we would do what we could. With regard to “no contractual requirement”, I mean, I’m not a PACE expert but, if witness statements form part of PACE and the continuation of evidence and support of evidence, then I can’t see how it couldn’t have been covered by the contract.
Mr Blake: In relation to your own statements, were they criminal proceedings, civil proceedings –
Jan Holmes: They were in support of whatever the Post Office were doing.
Mr Blake: And who asked you to provide those?
Jan Holmes: Well, as I say, a variety of people, I think. As I say, I had one request from a police sergeant who was doing an investigation into a post office. I did another one for Cleveleys, Cleveleys Post Office. I think I did one or two others as well.
Mr Blake: We’ll look at the Cleveleys example shortly.
Who internally was asking you to provide that statement; do you remember?
Jan Holmes: What, you mean in terms of Pathway?
Mr Blake: Yes.
Jan Holmes: Nobody. It was directly between me and the person requesting it from the other side.
Mr Blake: So, when it was the Post Office, for example, who would it have been that asked you to provide a statement?
Jan Holmes: It may have been Hilary, it may have been whoever was involved with that investigation. As I said, this is in the very early days when we were just understanding how this thing was going to work. So it was always on a case of best endeavours, and trying to meet what the customer wanted.
Mr Blake: We’ve heard –
Sir Wyn Williams: Is this just a discussion between you and the person, just someone approaches you, and says –
Jan Holmes: No, I’m sure I would have been looking for somebody with some authority. It wouldn’t just be anybody in the office.
Sir Wyn Williams: So they have a hunch that you might be the right person?
Jan Holmes: No. I did deal with individuals over time. Hilary Stewart was the latter one, taken over by Chris Paynter. There was another guy called Richard Cruise. I had some dealings with him but I think he subsequently left. So it wasn’t just a case of anybody could ring me up and say, “We need a witness statement.”
Sir Wyn Williams: So if I understand you correctly, persons who were known to you from working in conjunction with them over time and who would know that you would have the ability to provide this information?
Jan Holmes: I think that’s fair comment.
Mr Blake: Did you ever give a statement that addressed whether there were bugs or defects in Horizon?
Jan Holmes: No.
Mr Blake: We’ve heard about Gareth Jenkins. Did you know Gareth Jenkins?
Jan Holmes: Yes, I knew Gareth Jenkins.
Mr Blake: Were you aware that he provided statements in relation to criminal proceedings?
Jan Holmes: I am now, yes.
Mr Blake: Were you aware at the time?
Jan Holmes: No.
Mr Blake: Do you know why he might have been approached to give those statements?
Jan Holmes: Well, possibly because he was quite a knowledgeable person on the workings of the system. I’m not technical but, if I wanted some technical input to understand something about how the system was working, to be fair, Gareth was my go-to guy, because he knew a lot about a lot and so, you know, I made use of that knowledge as and when I needed to. So, if he was doing witness statements, I’m hardly surprised.
Mr Blake: Did you know somebody called Ann Chambers at all?
Jan Holmes: No. I remember reading the article, and I was thinking of somebody else called Ann on the project, but it wasn’t Ann Chambers. No, I didn’t even recognise the face.
Mr Blake: Were you aware of other people giving witness statements to support criminal prosecutions?
Jan Holmes: Yes. After I came back on to the project in 2001, the actual activity of data retrievals and provision of prosecution support had passed from audit, if you like, into the live customer support environment and was under the security banner. They had an operator there who was doing the retrievals, and she often had to provide a witness statement in support of the retrieval to explain how it was done and how the integrity of the data had been maintained.
Mr Blake: Do you remember who that was?
Jan Holmes: Not the off the top of my head, no, but I know where I can find out.
Mr Blake: Staying with this document before we go to the RFI procedure, if we could just scroll down, there’s a section on Transaction Processing and, if we go over the page, it says that, “TP [transaction processing] were also concerned at”, and then the first one on top of that page:
“The legal implications of manual alterations to computerised cash account by subpostmasters in the event of a court case.”
Is the ability to correct or erase data something that you were involved in at all from an ICL side from an audit side, for example?
Jan Holmes: Can I just see the context of that paragraph. I find that top bullet hard to accept. Okay, I can understand the first bullet. I was not aware at all that subpostmasters had the ability to carry out manual alterations to computerised cash accounts.
Mr Blake: And how about members of ICL?
Jan Holmes: I’m not sure about that because I’m not sure the level to which third-line support SSC got involved in manipulating – not manipulating, that’s the wrong word – in working the system in order to resolve issues and problems at that third level of support.
Mr Blake: Was it ever an issue that you had cause to investigate?
Jan Holmes: No.
Mr Blake: Can we look at POL00029165, please.
Just looking at the time, sir, this might be an appropriate time for a ten-minute break.
Sir Wyn Williams: By all means.
(3.01 pm)
(A short break)
(3.11 pm)
Mr Blake: Before the break we were looking at POL0029165. This is the Horizon System audit manual. Can you very briefly tell us what the purpose of this document was.
Jan Holmes: Yes, what this document set out to achieve was a description of the audit trails. It was like a further expansion of the audit trail functional spec. So this was trying to present information in a way that was understandable, legible and could be recognised and used by the internal audit community, whether that’s Pathway, Post Office or BA at the time.
So it looked at the requirements, and it looked at the roles, and it looked at what could be done, and then it went into describe what actually each piece of the audit trail consisted of.
Mr Blake: Thank you. It says there “Status, draft”, and it’s dated 17 January 2000. Are you able to assist us with that at all?
Jan Holmes: No, not really. I mean, it’s sort of – it went through a number of updates and drafting and improvements over time.
Mr Blake: I think version 1.0 was April 1999, if that assists.
Jan Holmes: It’s quite possible.
Mr Blake: We see there the name Chris Paynter POIA. Who was he?
Jan Holmes: Chris Paynter was at that time the single point of contact for data retrievals.
Mr Blake: What did POIA stand for?
Jan Holmes: Post Office Internal Audit.
Mr Blake: So was he from the Post Office?
Jan Holmes: Yes.
Mr Blake: Employed by the Post Office?
Jan Holmes: Yes.
Mr Blake: So this manual presumably was shared with the Post Office for a particular reason?
Jan Holmes: Yes.
Mr Blake: And what was that reason?
Jan Holmes: Knowledge sharing.
Mr Blake: Look at page 53, please, this is where the request for information is procedure set out. It says:
“POIA will request audit data via Request For Information form. This will contains a description, in business terms, of the times outlets, events, items and activities that the Auditors are interested in. This request has to be interpreted by Pathway Internal Audit and mapped on to the Audit Points and Files described earlier in this manual.”
The focus there is on auditors obtaining the information. Is that intentional?
Jan Holmes: Yes.
Mr Blake: Why is that?
Jan Holmes: Because that was – they were the audience for that data, the audit data, as far as we were concerned, through the request for information process.
Mr Blake: Was this route for obtaining information known by the Post Office?
Jan Holmes: I don’t know. I don’t know the answer to that because I don’t know what their internal communications were.
Mr Blake: So you have somebody from the Post Office being on the distribution list.
Jan Holmes: Yes.
Mr Blake: Are you aware, just from your experience of liaising with people from the Post Office, whether they knew that this was a route or the route to obtain information?
Jan Holmes: No, I don’t.
Mr Blake: If we go over the page, there’s a section on Investigation Support, and that says:
“The term ‘investigation’ is used in its broadest sense and does not limit itself to fraud. Any RFI is likely to be associated with a specific business event, eg an encashment, a bill payment, an outlet, a beneficiary. It is anticipated that the majority of this type will be based on the TMS journal, or will use it as a start point.”
Can you explain that to us, please.
Jan Holmes: Yes. I mean, I think 829 refers to prosecution support which implies perhaps some kind of fraudulent activity but, at the end of the day, this data was available for any kind of investigation that Post Office wanted to put it to. Fraud would be one of them or suspected fraud.
Mr Blake: Can we look at page 57 and the bottom section on page 57:
“Requesting audit data extractions. Prerequisite. Post Office Internal Audit will be expected to identify Auditors who are authorised to raise an RFI. It is not anticipated that this list will exceed two names. It is the responsibility of Post Office Internal Audit to notify Pathway Internal Audit of any changes to this list.”
What was the purpose behind that?
Jan Holmes: Looking back, I guess it’s to give Chris Paynter an opportunity to have some deputies, or perhaps to have people in different parts of the organisation who could request data with his authority.
Mr Blake: Why wouldn’t it be available to anybody?
Jan Holmes: Well, because we’d be swamped. If there was no control, then I couldn’t – well, I think we could have anticipated that there were masses of requests because people would – it’s like, if you suddenly find something that works, you exploit it.
Mr Blake: When you wrote this in ‘99/2000, what did you see as the role of the request for information in respect of criminal prosecutions?
Jan Holmes: It was just one of a number of reasons why that RFI might have been raised.
Mr Blake: So you did expect the RFI to be used for the purpose of obtaining that information?
Jan Holmes: Yes.
Mr Blake: But I think you said before that you didn’t expect the volume; is that right?
Jan Holmes: That’s correct, because in the early days the volume was, as we thought it – you know, as we saw it as being in the 10s/low 20s possibly a year, based on what we were being told.
Mr Blake: We’re going to move on to the particular disputes that I mentioned in 2000. Can we look at FUJ00121100, please. This is a letter from Keith Baines of the Post Office to Tony Oppenheim, and it’s dated 10 January, although 10 is crossed out and it says the 11th, and it has your name there amongst others. Do you remember this at all? I know you have seen it.
Jan Holmes: I remember the issue being around at the time, yes.
Mr Blake: Myles Blewett, was he a lawyer?
Jan Holmes: He was a lawyer with Masons.
Mr Blake: If we could scroll down, I’m just going to read a little bit for the record and to refresh your memory. First, halfway through the first paragraph:
“I understand the concern to be that, while Pathway are co-operating over ‘true’ audit access, it is being claimed that such provisions do not extend to supporting investigations for security purposes. Martyn has also suggested that if such access were to be granted, then a charge would be levied.”
The next paragraph starting the second sentence:
“Neither POCL nor the Pathway would wish to initiate such investigations without due cause (accepting that they may be time-consuming and distracting for both organisations), but nevertheless, where there are ground for suspicion, the issue must be investigated thoroughly. Without such an investigation we, jointly, could not be sure that the access controls and integrity of the Pathway services are not being breached.
“Apart from this practical perspective, I think that denying such access is at variance with” –
That’s the reference to the codified agreement there that we have already seen and the reference to the 18-month period and the requirement to support investigations.
Can we look over the page, please. At the bottom there it refers to 801.3, it’s a contractual term. Then it says:
“Pathway should provide access and may be paid for their assistance.”
Then there’s a holding response to that, and I’m not going to take you to it but it’s FUJ00121101. Instead I’m going to take you to a note from yourself at FUJ00058509, please.
Jan Holmes: Can I come back to that later?
Mr Blake: Yes, absolutely. Well, can you tell us in broad terms what’s going on there in that letter.
Jan Holmes: I think what was going on there is there was an issue that – there was a bit of confusion. As I said, the audit trail functional spec identified two particular types of audit trails, the commercial audit trail, which I think links into requirement 699 that talks about records, and I can’t remember the requirement are number that the operational audit trail, and operational audit trail, which were things like the TMS journal records and OBCS files all this stuff that was point on Legato.
Now, access to the records, commercial audit trail, was not done through the RFI process. That was invariably – if it was done it would be done through a direct request into the likes of Tony Oppenheim for access to the commercial records.
The RFI was for access to the operational audit trail and, I think, if you go back to that previous exhibit, there’s reference in there to the records under clause 801, and I think what’s happened is that there’s been at the conflation of two different topics into one.
Mr Blake: I think you mention that in fact in this document that we look at now which is a briefing note that you wrote.
If we look over the page, in fact, to page 3, please, I think the first page explains that you have received that letter and there’s going to be a discussion about it. Further down the page, please, under Statements, did you write this section?
Jan Holmes: Yes.
Mr Blake: Let’s look at paragraphs 2 to 4, please. It says:
“Arguably we already provide approximately 95 per cent of what they want via the TIP link on a daily basis.”
Now, is that similar to what Mr Folkes was saying in the document that I took you to earlier on?
Jan Holmes: I think it might be. I mean, I can’t remember for sure, but where my mind’s going on this is that a lot of this was to do with security investigation sort of doing fishing trips, what if, with the data, “Let’s have a look at this and see what there is, let’s have a look at this”, and not a case of, “We need this in order to progress a case or a fraud or an investigation.” I know there’s a subtle difference, but a lot of what we felt they required could be achieved through TIP.
Mr Blake: In fact, you refer to fishing trips in paragraph 3.
Jan Holmes: Yes.
Mr Blake: So you say:
“This could serve as their ‘fishing trips’ which I believe account for the few hundred queries anticipated by Bob Martin.”
Jan Holmes: Yes.
Mr Blake: “We would then extract from the audit archive to form the prime evidence for prosecution or dismissal. This approach was discussed with Hilary Stewart during the development period and Acceptance.”
So are you saying there that, if they want to look at the data for their own investigations, they had the TIP data, and it’s only when formal evidence is required that they should be coming to you?
Jan Holmes: That in effect is what I’m saying, yes.
Mr Blake: Can we go over the page, please:
“POCL have always maintained an organisational distinction between the Network Auditors who access the system that outlets and the Central Auditors who have access to historical audit data at the centre via Pathway Internal Audit.
“POCL have always been involved in development of the Audit Trail Functional Specification, the Audit Data Retrieval Requirements and the Horizon System Audit Manual where this separation has been reflected in the solution currently implemented.”
Is what you are saying in paragraph 6, about the document that I took you to before, where somebody from the Post Office was sent the specification and you’re saying there that the Post Office has always been involved in those discussions?
Jan Holmes: Indeed. I mean, the audit trail function spec was a CCD. So it’s a contract-controlled document. So it couldn’t be changed without the agreement of the Post Office.
Mr Blake: You have said at 7 that:
“The audit solution in its current form has been formally accepted by Post Office Counters Limited.”
And at paragraph 8 you talk about clause 801, which this point I think you were just making.
Jan Holmes: Yes, yes.
Mr Blake: Can we look at paragraph 11 –
Jan Holmes: I apologise for my intemperate language in there as well.
Mr Blake: No, not at all. Perhaps you can tell us about the atmosphere at that stage between yourselves and the Post Office, and why you may have made that comment.
Jan Holmes: I think it was just a throw-away line from me, “How can we be held responsible for a dodgy postmaster?” I mean, to be fair, in my role as pulling audit data for prosecution support, I had no idea or no knowledge at all of what level of underlying fraud was going on within the Post Office. I’m sure there was a level. There must be in a network that large with that many people. But how then that number would suddenly jump to 1 to 200 potential investigations a year, I just don’t know. It sort of wasn’t our job to query that with the Post Office. I would have expected them to have queried that themselves to try and understand better why there was this upsurge in perceived bad activity.
Mr Blake: Did you experience a sudden upsurge in requests?
Jan Holmes: It’s only when we got sort of requests like the Bob Martin’s 1 to 200, and you think: why is this number suddenly up where it is? We weren’t made aware of these volumes. In fact, when we talked to people like Hilary at the time and Jeff Robinson from the BA in the early days, it was 10s, 10s/20s, which we felt we could achieve within our current head count.
Mr Blake: Did you ever have a discussion with the Post Office as to why there was this sudden increase?
Jan Holmes: No, and that’s, I suppose, perhaps we could have done a bit more to question the increase. I would have half expected them to do their own investigation into that matter, and not rely on others to say: why have you got this increase in numbers?
Mr Blake: Can we go over the page, please, to page 5 and look at paragraphs 11 and 12. Paragraph 11, about halfway through that paragraph, it says:
“I do recall conversations with [and this is with Hilary] over a period of time where we discussed the potential for POCL using TIP data for their ‘fishing trips’ and then coming to me for the true data once they had progressed their investigation to a point of prosecution or dismissal. We never formally agreed that this would be the way to go but the potential was aired. Again though, note that it was her ‘regional audit colleagues’ and POCL Security Investigations.”
Can you tell us what you’re saying there.
Jan Holmes: Hilary was referring to the regional internal audit staff which she would have been dealing with, not POCL security investigations, which were a different part of the organisation.
Mr Blake: At some point did POCL security investigations become heavily involved?
Jan Holmes: Only at this point here. I think that happened because they suddenly realised they had lost FRMS.
Mr Blake: That’s what we talked about at the very beginning?
Jan Holmes: Yes.
Mr Blake: At the bottom of this page:
“This tells me that POCL knew that they were exposed as early as February 1999. The earlier discussions during the Audit Panel meetings were held with Fraud Risk Management, a separate part of Pathway to Audit. Bob Martin’s paid study was sponsored into Pathway by Fraud Risk Management.”
We’ve seen further correspondence, for the purposes of the transcript FUJ00121102, but it seems to crystallise in a letter of 22 February 2000 and I’m going to take you to that. That is FUJ00121103.
This is a letter from Martyn Bennett to Keith Baines. Can we scroll down slightly. I am just going to read to you two short passages. The first is the second paragraph, halfway through:
“In summary, Pathway maintain a transaction audit trail for the required period of 18 months and allow members of Post Office Internal Audit access to that audit trail in accordance with the agreed procedures and subject to known limitations which I described below.
“The issue we are concerned with has arisen for two reasons. First, a POCL organisation other than POIA, that is Post Office Network National Security Team, has requested access to the transaction audit trail to support security investigations. Second, PONNST recently indicated (in November last year) that it would require ‘a few hundred transaction/event logs to be provided during a full year’, which has never previously been raised with us, and Pathway does not consider it to be a contractual requirement.”
So do we understand there that the response from Pathway at this time was effectively, as you’ve said this afternoon, that it was a surprise to suddenly be receiving a few hundred requests?
Jan Holmes: Well, and from a group of people that we’d never heard of before.
Mr Blake: What was their job as far as you were aware?
Jan Holmes: National security team? I’ve no idea.
Mr Blake: There’s a response, and I don’t think we need to take you to it but for the purposes of the transcript FUJ00121104, and that’s the response from the Post Office that says it’s a question of legal interpretation and one for lawyers. I think you’ve seen that document.
But I’m going to take you to FUJ00121105, please. This is a letter from Martyn Bennett to Keith Baines and appears to record a tentative agreement on the issue.
Jan Holmes: Yes.
Mr Blake: If we could scroll down, this is 24 May 2000. It seems as though there’s a tentative agreement:
“ICL Pathway will provide up to 50 audit data extractions per annum for audit and security investigation purposes with a maximum of seven in any calendar month …
“Any additional extractions will be charged for on the basis that on average such extraction will take 1.5 man-days of senior consultant’s time (at £1,299 per man day) plus expenses. This we’ll cover all work including physical extraction of tapes at the Data Centre, loading onto machines, rebuilding the Correspondence server as necessary, searching for data, extraction and recording on to appropriate medium.”
Can you just tell us: why did it take so much time to obtain this information to extract the data?
Jan Holmes: It was just a complicated system. I mean, if you go into the Horizon System audit manual, I think at about section 10 there’s a description of the audit system including the retrieval activity. It was complex, and I can’t make any excuse for it being complex. It just was complex and, as such, it took a lot of manual effort to go through the process of identifying what – well, you had to interpret the RFI, you had to work out what files were needed to service the RFI, because we didn’t have a standard catalogue of RFIs; it was more a case of, “What do you want?”
We then had to interpret that to decide which files we had to get in order to service the RFI, and that meant that, when the Post Office got the data, they might have to extract information from two or three different sources on that data to get the information they wanted.
So there was no single report came out by extracting lots of different audit data files. So we had to do that. We had to locate where they were on the various Legato tapes which were the DLTs that we were using to store the data on.
Then we had to liaise with the data centres to get the tapes up and loaded onto the servers at the data centres so that the audit work station at Feltham could access them. There was only one audit work station in Feltham to work from, because that’s all we felt we needed, based on the original requirements.
It just took time.
Mr Blake: You told us earlier about a special fraud management system that would have been implemented for the Benefits Agency.
Jan Holmes: Yes.
Mr Blake: Is that what was missing in relation to the Post Office agreement, some sort of automated system?
Jan Holmes: Well, I was under the impression, and having read through some of the documentation, that the Post Office were either planning to or were in some kind of agreement with the Benefits Agency to make use of some aspects of FRMS, because obviously if there were benefit encashment issues going on, not only would the Benefits Agency need to know, but also the Post Office would need to know that there may have been collusion going on between offices between postmasters and beneficiaries. I don’t know.
So I think there was some kind of fledgling agreement going on to give the Post Office access to the FRMS for certain types of data searching and information retrieval, but I don’t know.
Of course when that went, there was nothing.
Mr Blake: So the period of time you’re talking about at the moment was some way back, was it?
Jan Holmes: Yes. I mean, BA dropped out of the contract in 1999, I think, just as we were approaching acceptance. Of course, when they dropped at all of their services dropped out with them including the FRMS.
Mr Blake: Point 3 on this letter says:
“All requests for extractions will continue to be channelled through ICL Pathway’s audit manager who is now Brian Mooney.”
That suggests that by that stage, May 2000, you had moved?
Jan Holmes: I’d moved on.
Mr Blake: Now, in your experience going forward from the date especially from when you rejoined, were the Post Office reluctant to obtain audit data?
Jan Holmes: I don’t know, because the responsibility for delivering that service had moved from audit to customer services. Security had taken it under their umbrella, and security at that time were working for customer services as part of the live service delivery.
Mr Blake: Are you aware of that tentative agreement, whether that was the final agreement, whether there was a similar agreement?
Jan Holmes: No.
Mr Blake: Were you aware of a significant sum along those lines being charged for audit data?
Jan Holmes: No.
Mr Blake: So that was no longer part of your remit?
Jan Holmes: No, it was – yes, it was no longer part of my remit.
Mr Blake: Can we go to FUJ00001419, please. Can I ask what this document is.
Jan Holmes: The SADD. Not really. There was a lot of information in there that I recognise but in terms of the SADD itself …
Mr Blake: You will be relieved to know I am not going to take through the various complicated diagrams. I’m just going to take you to page 4, please. This talks about fraud risk management. Can we look at the bottom of this page. It says:
“Pathway’s strategy is to identify high-risk situations and adapt systems as necessary to:
“Minimise fraud exposure within the Pathway solution.
“Provide information service to POCL to aid fraud investigation and to minimise fraud.
“The information provided is:
“Information to aid the investigation of actual fraud incidents [and then over the page]
“Certification relevant to operation of the system as required by PACE”, et cetera.
Then it’s these two I am particularly interested in:
“Information for the investigation of system boundary-related incidents and trends; for example, counter staff-related fraud with the aim of developing improved procedures.
“Analysis of incidents and trends within Pathway’s immediate control, to improve its systems.”
Was this something within your remit or was that outside?
Jan Holmes: No, it was outside.
Mr Blake: What I’d like to understand is were you aware of ICL Pathway having any proactive role in identifying high-risk situations, or providing analysis of incidents and trends relating to, for example, criminal prosecutions?
Jan Holmes: No. I don’t think I was. In fact, I wasn’t.
Mr Blake: You weren’t aware? Do you think that they happened or didn’t happen?
Jan Holmes: I don’t know. I wouldn’t know.
Mr Blake: We looked at the sophisticated tools that were proposed in that BA system, for example, the fraud risk management server. Do you think that the lack of analysis or the lack of a special system along the lines that the Benefits Agency had proposed was a missed opportunity?
Jan Holmes: For the Post Office?
Mr Blake: Yes.
Jan Holmes: Well, yes, I guess anything that would improve their operation must be to their benefit. So, if they opted or chose not to do something that may have benefited from them, then I’m not sure what they would expect Pathway to do to fill in.
Mr Blake: Were you ever aware of any serious analysis of incidents and trends arising from that kind of data taking place at ICL Pathway?
Jan Holmes: No.
Mr Blake: I’m going to move on to the Cleveleys case. Can we look at FUJ00080715, please. Now this was a document that you produced, and it’s a report on Cleveleys Post Office. Do you recall that a subpostmistress was dismissed, I think, very early on November 2000?
Jan Holmes: Yes. It didn’t come to our attention for witness statements or support until 2004.
Mr Blake: And that’s something we will come to because obviously that’s relevant for the retention of the audit data.
Jan Holmes: Mm-hm.
Mr Blake: Now, you would accept that, from what we’ve discussed this afternoon, in that period, so November 2000, there were undoubtedly issues with the Horizon System?
Jan Holmes: Yes, I do know now that that was the case. At the time I didn’t.
Mr Blake: There was in this particular case a jointly appointed expert who produced a report that said that Horizon could have caused an error. Are you aware of that?
Jan Holmes: Who was the expert?
Mr Blake: If we scroll down, I think this document summarises the fact that there was an –
Jan Holmes: Oh, are you talking about his –
Mr Blake: Yes.
Jan Holmes: Sorry, there was an expert.
Mr Blake: And they produced a report that said that Horizon could have been responsible.
Jan Holmes: Yes. They also accused us of sending incomplete data as well, but that’s beside the point.
Mr Blake: Can we look at page 4, please, and if we could scroll down. Can you tell thus purpose of this particular report.
Jan Holmes: This was me just capturing the after-event story, if you like, because it actually proved to be a nonevent from our point of view. I actually got as far as the court house before it was cancelled, because there was an out-of-court settlement reached. So I felt it was appropriate to capture what had happened, because there’s quite a lot of my time and the company’s time had gone into supporting this prosecution, and it just disappeared.
Mr Blake: If we look at Scope, it says:
“This report does not set out to address the case itself, merely what POA have provided as support and some of the issues identified in providing that assistance. It does make recommendations as to how future involvement by POA in these non-standard cases might be better managed.”
Were you intentionally not going into the rights and wrongs of a particular case?
Jan Holmes: I can’t; I’m not qualified to do it.
Mr Blake: Were you instructed to do this at all, to carry this out by anybody?
Jan Holmes: Well, probably yes, I wouldn’t have just done it, you know, voluntarily. I suspect I was probably instructed by Martyn to look after this, or perhaps security asked me to look at it.
Mr Blake: Can we scroll down, please. There’s a section there talking about the expert report. Could we keep on scrolling, please. Sorry, if we could just go up slightly. Thank you.
At the bottom just above 3.2, it says there:
“We have offered to host the expert at any of our locations so he can analyse [the] data directly, speak to the experts and a walk-through of the problem management cycle for himself. He will not have seen this offer since it was contained in the email that accompanied our final response, and this has not been passed on to the Expert pending an outcome of an out-of-court settlement offer by POL to the PM.”
Then it’s got an update there, I think, or 1 April 2004:
“Confirmed with Jim Cruise that the recommended three months’ salary offer is now with POL for approval.”
Were you aware at the time that they were making the settlement offer?
Jan Holmes: I don’t think I knew about that until we were in court, until I had actually turned up.
Mr Blake: If we could scroll down to the chronology, if we look that bottom entry there. It’s 6 February 2004. It says:
“Nothing more was heard for almost six months until we received a letter from the Post Office containing the Expert’s report. Post Office were concerned that the report claimed that the equipment installed at Cleveleys was defective and that the Horizon System Helpdesk was more concerned with closing calls than resolving problems. Post Office feared that, if the report went unchallenged, it could set a precedent for other cases being progressed against postmasters.”
Is that something you remember at all, the fear or a precedent being set, for example?
Jan Holmes: I don’t deny writing that chronology, and that Post Office fearing was probably a collective concern that was expressed by people like myself, Graham Hooper, who was the security manager, and possibly Martyn Bennett as well.
Mr Blake: I think it says there the Post Office feared that, if the report went unchallenged, it could set a precedent. Were you aware from the Post Office of those concerns?
Jan Holmes: Oh, I’m with you, yes. Sorry, I led you down a garden path there. Yes, if that’s Post Office – because I can understand we would be concerned at that as well, but that probably came through Jim Cruise.
Mr Blake: Do you remember it at all? Do you remember any fears being communicated to you about –
Jan Holmes: Not specific fears, no.
Mr Blake: I think it mentioned somewhere in this report but slightly higher up, that they hadn’t previously found a problem, so that could be a basis for saying that the system was functioning sufficiently – sorry, I don’t have the paper copy of this one. If we go up a little more:
“POA cannot prove, in the literal sense, that the system operated correctly during 2000 since we do not have the transaction data that will demonstrate that fact.”
Just pausing there, you didn’t have the transaction data, I think, because the 18-month period had passed; is that right?
Jan Holmes: Yes.
Mr Blake: As you said, you were only contacted after that period had expired about this particular case.
“Equally, any proving that we could do, for example, by design walkthroughs with the Expert, would prove nothing since it would be a 2004 system baseline that was being considered, not one from 2000. We can infer that it was since we are not aware of any contemporaneous POL prosecutions failing due to the system not operating correctly.”
So was the feeling there that, because prosecutions hadn’t previously failed, that would be sufficient to infer that there wasn’t a problem?
Jan Holmes: I don’t think that’s an unreasonable position to take. You know, if there is no history of prosecutions of that type, there is nothing to suggest, to my mind anyway, that there would be a problem now.
Mr Blake: Looking at that and then turning the page to the bottom of the next page and that section that I took you to a moment ago about the fear of setting a precedent, is your recollection one of concern that a successful, or an unsuccessful case, in this particular case, the Cleveleys case, might set a precedent undermining future prosecutions?
Jan Holmes: I think, reading that now, that looks to me that it took six months before we received a copy from the Post Office containing the expert’s report and, yes, you’re right that Post Office, not Post Office account, and if they were concerned that the report claimed that the equipment was defective, then that could be seized by other people and said that, if it’s defective there, it’s defective elsewhere. If the Horizon system help desk was more concerned with closing calls and reserving problems, then that could be seized upon and used again. So the Post Office were looking for us to challenge that report, which we did.
Mr Blake: Can we go to the bottom of page 6, please. It’s the final entry, 13 August 2004:
“Notified by POL that Post Office had made an increased offer to [the postmistress] to drop the case and this had been accepted. Not required in court.”
We’ve seen a document – I’ll give you the reference but I don’t think we need to go to. It’s POL00090575 that says that the settlement was for £187,000 which included costs. Were you aware of the figure at all?
Jan Holmes: No.
Mr Blake: Were you aware – I think you said that you only found out on the day about settlement. Did you go to court or –
Jan Holmes: I did, yes.
Mr Blake: Were you aware that it was a substantial figure?
Jan Holmes: No.
Mr Blake: Did anyone express any views as to how it was settled and –
Jan Holmes: No, I’m quite frankly stunned at the amount because, at the end of the day, the lady removed the equipment from the office and refused to return it. So quite why she was getting compensation beats me.
Sir Wyn Williams: Well, if she had a claim for unfair dismissal, that’s what the counterclaim was about.
Mr Blake: We know that shortly after this incident we have the Lee Castleton case. Was that case you were aware of at all?
Jan Holmes: No.
Mr Blake: Do you think this case was a missed opportunity in getting to the bottom of problems with Horizon?
Jan Holmes: What, Cleveleys?
Mr Blake: Yes.
Jan Holmes: Yes, I mean, possibly so, yes.
Mr Blake: I’ve got a few small further topics to ask on behalf of Core Participants. Can we look at POL00089802, please. This is a November 2001 audit of customer service support processes. Can we look at page 4, please. This was written by you or originated from you.
Jan Holmes: Yes.
Mr Blake: Do you remember the purpose of this document?
Jan Holmes: Well, it was an audit report, wasn’t it?
Mr Blake: What did it address?
Jan Holmes: The processes that were being used by customer services as part of their role.
Mr Blake: Can we go to page 5, please, and there’s an overall opinion there. Can we scroll down a little bit. Thank you very much. I’m just going to read that first couple there. It says:
“There are a number of relatively minor issues that, while not impairing the current management of incidents and problems, could, if accepted and addressed, improve the performance of this part of CS.”
CS being customer support –
Jan Holmes: Customer services.
Mr Blake: “Provide a definition and guidance for when an incident should be escalated to become a problem.
“Introduce formal root cause analysis into problem and complaints management as a matter of course.”
If we go over the page, please, to the bottom there’s discussion – do you remember this – about it not being easy to identify what constitutes a problem and when an incident becomes a problem. Is that something you remember at all?
Jan Holmes: Yes, vaguely, yes.
Mr Blake: There’s the box after that on the next page, which says:
“Although relatively trivial, the lack of guidance or definition can introduce uncertainty and the opportunity for ‘problems’ to be missed or unnecessarily escalated. It is recommended that [there’s an update] to provide definition criteria and, if considered useful, examples.”
That’s question that I’m asked to ask you is, whether you still looking back think that that was a trivial problem.
Jan Holmes: The problem in itself wasn’t trivial because, at the end of the day, there were some rules around it but they weren’t formalised, as far as I recall. So formalising was a trivial bit. The actual escalation from an incident to a problem is not in itself trivial, because it generates a whole different set of actions. So I think what I may have been referring to as trivial is the lack of guidance.
Mr Blake: Thank you. We can take that down, thank you.
I’ve got a question about the help desk. At paragraph 11B of your statement you point to two documents which set out the adequacy or otherwise of the help desk. We’re going to explore this in a lot more detail during Phase 3, but can you tell us your broad views about how effective the help desk was, please.
Jan Holmes: Which help desk? Sorry, I’m saying that because there are a number of different levels of help desks. There were the help desks that sat within the Post Office environment that were looking after advice and guidance, because that was all being handled at one time by the Horizon System help desk and then it got separated out. I think that might be what the EDSC was about. That was up in Barnsley, I think, because I went up there once for a visit.
The Horizon System help desk was then the next level. But, I mean, I didn’t have a particular view on it. It was there to fulfil a function. Again I did visit and it was split over two sites. I did visit both sites as part of an audit, but it seemed to be functioning in the sense of calls being handled and sufficient manpower to do the work. The audit didn’t involve going in to look at specific calls and saying, “You handled that well”, or, “You didn’t handle that well.” That’s not what the nature of the audit was.
Mr Blake: Finally, as someone who audited the system and was involved in discussions about retention periods and the provision of data for prosecutions, what is your view about the Post Office relying on Horizon data to prosecute people?
Jan Holmes: Well, in the olden days, I guess, when the Post Office had paper records in the outlet, they would rely on those records to prove guilt or otherwise – not guilt, sorry, to prove that something was going on or not. Those paper records disappeared when Horizon came on the scene. So they had to place reliance on Horizon data then, to carry at those investigations and to make the necessary prosecutions, if that’s what they wanted to do.
So they had no option but to rely on the Horizon data.
Mr Blake: Knowing what you know about problems with Horizon data, what is your opinion?
Jan Holmes: Well, they still had no option. Yes, I mean, what’s happened is disastrous, it’s very bad, but at the time it wasn’t known and, you know, I’m not sure what I can say to that.
Mr Blake: When you say it wasn’t known, it was known –
Jan Holmes: It wasn’t known to me; let’s put it that way. Whether it was known elsewhere, I don’t know.
Mr Blake: When you say it wasn’t known, you mean nobody connected the prosecutions with the problems with the Horizon System, or was the something else that wasn’t known?
Jan Holmes: No, as I alluded to earlier, I would have expected the Post Office to have made that connection somehow, because we sort of couldn’t. All we could do is, we were taking requests for data. We didn’t know what they were doing with it. They never said, “This data is required for a potential prosecution.” It was just, “This data is required.” So we didn’t know what they were doing with it.
They did and, you know, I think it was beholden on them to have been, I don’t know, more careful – is that the right term – but to be, you know, had a duty of care to make sure they were using that data in an appropriate manner, and that it was correct, which sounds odd because you might think that was our job to make sure that’s correct. What we were able to do was demonstrate that we collected what was sent to be collected. We stored it securely. When we retrieved it, we proved that we’d stored it correctly by recalculating checks and seals, and it was extracted and distributed in a controlled environment.
Mr Blake: You had identified problems with Horizon data in 1998 and 1999 to management within ICL, and you were aware and management were aware that there was a dramatic increase in requests for data from the Post Office. Do you think somebody should have put those two together?
Jan Holmes: Sorry. When you say we were aware that there were problems with the data?
Mr Blake: Well, for example, the EPOSS PinICL Task Force and the various reports I took you to first thing today identified that there were underlying system issues. There was severe criticism from – we’ve heard from Mr McDonnell, for example. Do you think somebody, I don’t know who, and you can perhaps tell us who should have put those two things together?
Jan Holmes: No. I can’t offer a name or a group who might have done that.
Mr Blake: You may not be able to offer a name or a group who might have done that but who should have done that?
Jan Holmes: Within Pathway?
Mr Blake: Yes.
Jan Holmes: I really don’t know. I really don’t know.
Mr Blake: Thank you very much, Mr Holmes. I don’t have any further questions. There may be some questions from other Core Participants. Mr Henry looks like he is consulting his papers.
Sir Wyn Williams: No, he has no questions.
Questioned by Mr Henry
Mr Henry: Thank you, sir.
Mr Holmes, would you please be kind enough to look at FUJ00080690, one of the areas on which I have been permitted to ask you some questions. We can see, if we may, over the page at page 2 – thank you very much, Document History, 18 September ‘98, Initial Draft Following Task Force Completion. This was written with Mr McDonnell, wasn’t it?
Jan Holmes: Yes, that’s right.
Mr Henry: Then 14 May ‘01 raised to V1 or VI.0 administrative catch up?
Jan Holmes: It’s V1. Yes, I think that was just me.
Mr Henry: That was you because you were back by that time because, although you had left, you came back in 2001?
Jan Holmes: Yes, actually I did, but it was after this date. So I’m struggling now …
Mr Henry: Because you had gone to a Prosell or something?
Jan Holmes: I’d gone to Propel and then I’d come back after about a year because that programme collapsed.
Mr Henry: Absolutely, as you stated in paragraph 3 of your statement. Could we go to page 4 of 20, please, and could we go to Management Summary, please. We can see just above that in Scope that we’re dealing with a period there of EPOSS between 19 August and 18 September, and that of course would have been in 1998.
Jan Holmes: Yes.
Mr Henry: Then we have the Management Summary which – let’s take that as read. Could we go to page 7 of 20, please, and we can see there:
“It is clear that senior members of the Task Force are extremely concerned about the quality of code in the EPOSS product”, et cetera.
Then over the page to 8 of 20, please and at 4.2 we can see at letters a to f that the PinICL stacks in relation to this concerning EPOSS have been divided into six separate pots.
Jan Holmes: Yes.
Mr Henry: So no doubt that was in order to sort of track the matter chronologically, but also it was there because perhaps you were, for example, doing a holding stack while fix activity was underway, post fix but pre-link test.
Jan Holmes: Yes. That was to allow us to do a bit better analysis of where these PinICLs were in the fix cycle because – sorry – don’t forget that this was an extra piece of work that was sitting on top of the existing activity that was going on. So, while the Task Force was looking at this wedge of PinICLs, EPOSS was still going on in the background. That was the point that Dave made this morning.
Mr Henry: Of course.
Jan Holmes: So we were trying to empty the swamp while the tap was still on at the top, and it was a virtually impossible task.
Mr Henry: Exactly, exactly. That’s why I want to ask you why you’re coming back to these on 14 May 2001.
Jan Holmes: It was just an admin clear-up, I think, of documents that were still sat at draft.
Mr Henry: You see, because it coincides exactly with the date of a Post Office board meeting on the very same day, 14 May 2001, RMG00000009, which speaks of error, that’s error reconciliation activity, and Horizon Issues.
Was this report of yours with its dismal observations on EPOSS retrieved because of information that had been received from the Post Office at that time?
Jan Holmes: Not to my knowledge, no.
Mr Henry: So the coincidence in this case –
Jan Holmes: It is coincidence.
Mr Henry: Just a coincidence, right. Could I just ask you, please, to address matters – again I have permission for this, but address matters as to who was going to pay for RFIs in the event of prosecutions. We’ve seen a document put up on screen by Mr Blake that Mr Martyn was suggesting that there might be a few hundred per annum.
Jan Holmes: Mm-hm.
Mr Henry: You expected, you said, POCL to query the upsurge themselves.
Jan Holmes: I would have thought that was a prudent thing to do.
Mr Henry: You were asked by Mr Blake, “Did you ever ask POCL why there was this sudden increase”, and then you said, “I suppose we should have asked why this increase in numbers.” Do you remember saying that?
Jan Holmes: Yes, yes.
Mr Henry: You said to Mr Blake a very short while ago that the Post Office really had no option but to use Horizon in the prosecution of its subpostmasters.
Jan Holmes: Based on my understanding that the evidence of activities in the outlet was captured by the system, yes.
Mr Henry: So, therefore, if it was demonstrated to be manifestly unreliable in certain respects, and it might only have been 1 per cent or a fraction of 1 per cent, this whole edifice would have come down in an enormous crash, wouldn’t it?
Jan Holmes: If that was the case, then yes, I guess it would have done.
Mr Henry: Of course it would, because obviously it would have been a matter of legitimate public interest. So what I’m going to suggest is that, whether you were sighted on this or not, it was not in ICL Fujitsu’s best interests to ask that question, was it?
Jan Holmes: Well, that’s an interesting one, as you might expect me to say. Was it in our best interest to ask about the integrity of the system as demonstrated by the high number of prosecutions that were going on? I don’t even know how many prosecutions were going on. But the high number of investigation requests that were being floated by Bob Martin suggests that there was a lot going on in the background that we didn’t know about.
Would it have been in our best interests? I guess, in the interests of honesty and integrity, then yes, it would be.
Mr Henry: It would have been disastrous reputationally for everyone concerned; even if it was only 1 per cent or a fraction of 1 per cent, your system was manifestly unreliable.
Jan Holmes: Well, yes, it would be, yes.
Mr Henry: We can if we need to – the counsel to the Inquiry have been notified about various documents, but a defensive attitude, you would agree, has been adopted or was adopted by ICL in respect of criticisms of Horizon?
Jan Holmes: Where was that evidenced?
Mr Henry: Well, I mean, for example, in the Cleveleys issue where you were, as it were, deriding the opinions of the expert on behalf of the subpostmistress in various negotiations with POCL. There was a defensive attitude as to system error and unreliability, wasn’t there?
Jan Holmes: Can we get that document back up, please. I’m not happy with that.
Sir Wyn Williams: Well, I think the document speaks for itself, Mr Henry.
Mr Henry: Yes, it does.
Sir Wyn Williams: I can read what it says.
Jan Holmes: Yes, but I’m not happy of the implications of what the gentleman was saying.
Sir Wyn Williams: I think Mr Henry is referring to one part which stuck in my mind where criticisms – I’m using that word in the general sense – were being made of the quality of the expert evidence. That’s it, isn’t it?
Mr Henry: That’s right, sir.
Jan Holmes: I’m sorry to carp on about this.
Sir Wyn Williams: Bring it up then. Bring up the document.
Jan Holmes: I don’t think it’s warranted –
(Unclear: simultaneous speakers)
Sir Wyn Williams: I don’t think it’s being suggested or I’m not suggesting it, if you think I am, that you were making up those criticisms.
Jan Holmes: No, I just didn’t like the tone.
Sir Wyn Williams: Oh, I see. All right.
Mr Henry: I don’t need to go to it, sir, in the light of your supervisions. It is only if the gentleman would like to.
Sir Wyn Williams: I got the point you are both making. Let’s put it like that.
Mr Henry: I just want to return to that document that you, as it were, had an administrative catch up about very quickly – no need to put it on the screen – 14 May 2001, because, very shortly or contemporaneously with that, one of the Core Participants I represent fell under suspicion, was subsequently prosecuted and went to prison the following year, a 19-year old girl sent to Holloway.
Dealing with requests for unused material or material RFIs, as you say, you were expressing concern about “dodgy postmasters”, and I realise that you –
Jan Holmes: An unfortunate term.
Mr Henry: Of course, I don’t refer to that pejoratively. You have apologised for the terminology. It was your understanding at the time. But you were told that roughly it was going to be about 200 a year.
Jan Holmes: Mm-hm.
Mr Henry: Eventually on 24 May 2000 a compromise was reached, that 50 queries would fall within the contract and then everything else it would cost about £1,500 per day; do you remember?
Jan Holmes: Yes, I’ve seen the letters.
Mr Henry: When that matter came to trial, Ms Felstead’s trial, a request was made by your company, that you then worked for, for £20,000 of the defence for Ms Felstead in relation to the provision of information. That wouldn’t surprise you in the light of what had happened, would it?
Jan Holmes: When did this happen?
Mr Henry: That happened – well, she went to prison in August 2002. So I’m referring to the period of obviously between May 2000 and August 2002. May 2000 the agreement, prosecution in 2001-2002, sent to prison August 2002. That wouldn’t surprise you, would it?
Jan Holmes: Look, I’m not trying to duck the issue, but I wasn’t on the project then and when I came back, it wasn’t my area to be responsible for.
Mr Henry: You came back, however, in 2001?
Jan Holmes: Yes, but it didn’t – the role then didn’t include the provision of audit data to the Post Office. That had been passed to customer services.
Mr Henry: I see. So, if that wasn’t your role, why are you therefore retrieving that report that I put to you that very beginning of my questions?
Jan Holmes: Cleveleys?
Mr Henry: Not Cleveleys. I’m talking about Ms Felstead, talking about, if that wasn’t your role, why were you as an administrative catch-up, referring to a report that you and Mr McDonnell had written in 1998 in May 2001?
Jan Holmes: It was just that an administrative catch-up; that was all. There was nothing – there’s nothing – no ulterior motive behind doing that.
Mr Henry: What I want to suggest to you is that clearly, obviously the problems in EPOSS hadn’t gone away, which is why you were pulling up that report for administrative catch-up on 14 May 2001.
Jan Holmes: That’s just raising it to version 1 from –
Mr Henry: Sorry?
Jan Holmes: That was just raising it to version 1 from version 0.1. There was no change to content.
Mr Henry: I see.
Jan Holmes: I’m sorry, I’m not quite sure what point you are trying to get at here.
Mr Henry: What I am trying to say is obviously the problems with EPOSS referred to therein had not been solved.
Jan Holmes: Clearly not.
Mr Henry: Yes. I’m grateful. Could I ask you please to just consider this. You may not have been familiar. I mean, obviously you mention in your statement that this was a massive exercise, and it was the largest nonmilitary exercise in IT in Europe at that time.
Jan Holmes: I believe so, yes.
Mr Henry: Mr Martin is asking for about 200 cases a year. There were 19,000 post offices roughly at about that time. 1 per cent by my ‘O’ Level Maths would come to 190, wouldn’t it?
Jan Holmes: Yes.
Mr Henry: So you think that reflects upon, as it were, the percentage error that was clearly by that stage unresolved?
Jan Holmes: I’m sorry, I’m not following you.
Mr Henry: Well, the system bugs, errors and defects.
Jan Holmes: What’s the number of post offices got to do with that?
Mr Henry: Well, just the fact that, if you have a 1 per cent error rate, you have got 19,000 post offices, you might get 190 referrals per annum for that reason.
Jan Holmes: But the requirements for the retrieval process were based on what we were told at the time by the Post Office Internal Audit and the BA internal audit who were the authorised users of audit data. Bob Martin’s group were not on the horizon at all. I mean, you are probably not wrong in your 190 but, at the time when we were putting together the requirements and specifying the system, it was based on what the customer said they wanted.
Mr Henry: Or not necessarily – and this is my final question, sir, and again it’s a matter upon which you or counsel to the Inquiry has given permission – not necessarily what the customer wanted, and I’m now moving to another subject, but what you were prepared to give them. In other words, an expert who could, as it were, give a sort of declaratory statement that the system was working properly. “If you want to check it, it’s going to cost you this amount of money”, but essentially somebody like Mr Jenkins to come and, as it were, speak in a lapidary fashion that there’s nothing wrong with this.
Jan Holmes: I can’t answer to that, can I? It’s nothing to do with me. I wasn’t involved.
Sir Wyn Williams: I think that is a very long comment, not a question, Mr Henry.
Mr Henry: Forgive me, sir. Final issue, sir, final issue, and I promise you this and I thank you for your indulgence.
You are not a techy, sir?
Jan Holmes: No.
Mr Henry: No. And Mr Austin, your friend, appointed you?
Jan Holmes: Yes.
Mr Henry: At the time, Mr Gareth Jenkins was your go-to guy for things that were technical?
Jan Holmes: If I wanted to know something, yes.
Mr Henry: He doesn’t appear to be that visible or prominent in the materials that were provided to you or the Inquiry. Can you think of any reason for that?
Jan Holmes: No, not really. I mean, my involvement with Gareth was when I wanted it.
Mr Henry: Could I ask you please, in conclusion, to go to FUJ00078284 and could we please go to page 23 of 32. I just observe it’s December 1998 and the date is 15 January 1998, but I think that must be ‘99 – that there was a typo there but it’s immaterial. Can we go to “live trials”:
“Deliverables for live trials [et cetera, et cetera] high level document extraction and filter document produced by Michelle Myles …”
You, sir, are mentioned. I’m not sure if you actually saw this:
“… who specified the requirements for audit has confirmed that he does not have a requirement for GUI interfaces to any of these tools. He only requires batch tools. He may at a later date, after the process has been used for some time, ask for such enhancements. However, it was agreed at a meeting [et cetera, et cetera] that the requirement for GUI interfaces would remain …” et cetera, et cetera.
Can you now remember what “GUI” stands for?
Jan Holmes: Graphic user interface.
Mr Henry: Exactly. They are more user friendly for the user aren’t they?
Jan Holmes: Yes, they are if you’re going to start looking at the data and whacking it up on your screen, so you can get an immediate feed of what that data is telling you off the record.
Mr Henry: Absolutely, and was this for the end user, a GUI interface for the end user, i.e. the subpostmaster, or for somebody doing your audit?
Jan Holmes: No, the subpostmaster would not have access to this data.
Mr Henry: So, in other words, the GUI which would have been – allow you to get up more data, allow you to see it more clearly, be more analytical, that was, as it were, dismissed in preference for a much less sensitive batch tool?
Jan Holmes: Because that was what was needed at the time. That was what was being – that was what we were told was required at the time.
Mr Henry: I see. But in view of the first – in view of the answer you gave me, GUI would have been more sensitive to detecting bugs, errors and defects, wouldn’t it?
Jan Holmes: No, it would have just made the data easier to read on the screen, that’s all. I mean, the data – sorry, the data is still there. It’s up to the users of the data to investigate and interpret it.
Mr Henry: Well, thank you very, very much. Nothing further, sir. Thank you.
Sir Wyn Williams: Thank you. Anyone else?
Mr Holmes, during the course of answers you gave to Mr Blake, I got the impression – and if I’m wrong, please correct me – that you’ve retained some of the documents over all these years.
Jan Holmes: Yes, I have.
Sir Wyn Williams: Have you retained any of your witness statements you’ve made?
Jan Holmes: Probably in the midst of the archive, yes.
Sir Wyn Williams: I was going to ask you, if you would please, to dig them out and send them to the Inquiry.
Jan Holmes: Yes, I’ll do that.
Sir Wyn Williams: We’ll send you a formal letter to the effect, all right.
Jan Holmes: Who do I send it to?
Sir Wyn Williams: Oh, you’ll be given the details.
Jan Holmes: Oh, all right, fine. Yes. I can dig out what I’ve still got.
Sir Wyn Williams: Yes, that’s good. Thanks very much. Thank you very much for your witness statement and thank you for coming to give evidence to the Inquiry.
Jan Holmes: You’re welcome.
Sir Wyn Williams: Is that it, Mr Blake?
Mr Blake: That is, sir. We have Mr Cipione returning tomorrow at 10.00.
Sir Wyn Williams: Jolly good. So 10.00 tomorrow morning it is.
(4.22 pm)
(Adjourned until 10.00 am the following day)