21 October 2022 – Paul Rich and Peter Copping
Hide video Show video
(10.00 am)
Paul Rich
Questioned by Mr Blake (continued)
Sir Wyn Williams: I can hear what goes on when the usher makes her provisional announcements but I didn’t hear the fire alarm. There we are.
Mr Blake: Mr Rich is ready.
Yesterday we spoke about the evaluation board and ComPEC and MaPEC. I’m going start today in early 1996 and can we look at POL00031278, please. That’s an email to you of 10 January 1996. Can we look at the top half of that page, please.
There is a meeting with Cardlink. Basil Shall, and Wendy Powney met with Cardlink. Who are they?
Paul Rich: They are the – I think they’re IT people and I think they’re POCL IT people. I know Wendy well because eventually she worked for me but not at this time.
Mr Blake: That says:
“… it became quite clear that our requirements process is lacking in 3 major areas:
“Reconciliation/Role of TIP
“Office Balance/Cash accounting process.
“Interfaces with the TMS …
“If we don’t give them direction (which they are desperately seeking) they are very likely to choose an option which is not in our best interests.”
Do you know who that direction was to: “if we don’t give them the direction”? Who would you be directing?
Paul Rich: I don’t think it’s for me to direct them. I think this is me – them coming to me because I’m part of the Evaluation Board in my role as partnership development person, then working to Mr Peaple, and they’re saying – I think I explained yesterday we allowed the group IT people and our own IT people access to the suppliers as the bids were evolving so that – although they weren’t part of the formal procurement process and, therefore, didn’t form part of the value assessment factor group because they weren’t at sufficient arm’s length to do that, whereas I think John Meagher and, I think that must be Jeremy Foulkes, are. I think they are part of that team.
I think what he was doing was asking me to co-ordinate within the business the relevant experts in that field to try to clarify some of the reconciliation cash accounting interface, the TMS. I can’t remember what TMS stands. Role of TIP is transaction information processing.
I think – I can’t remember the date. This is January ‘96. I think you’ll find in the bundles notes from or to me from relevant financial experts within the business around those reconciliation and accounting issues. So I was sort of promulgating the actions so that his concerns would be addressed.
Mr Blake: Absolutely. So when we look at the bottom of this page it seems as though the affected area seems to be commercial. So you’re looking at it from a commercial perspective at that stage, in relation to those issues, rather than technical; is that right or have I misunderstood?
Paul Rich: Yes, I don’t think that’s quite right. This is a memo to me, isn’t it, if you just go back to the top of it.
Mr Blake: It is.
Paul Rich: So it’s not from me. So:
“An early commercial ‘must have’ supported by Charterhouse … did not control there end-to-end client product.
“This was to ensure … could not cut POCL out of the transaction.”
Yes, I understand. Charterhouse were reviewing the overall financial security and probity of the whole thing for us. There are other documents that relate to them, I think, in ComPEC and MaPEC papers, with their input, and I think this is really more about what the service specification requirement really was, the SSRs as they became known, which were fed in both from ourselves and from DSS and Benefits Agency to the – I don’t know if it was five suppliers or three suppliers at that stage.
Mr Blake: If we look at the bottom right-hand corner and those two bullet points, the top one is:
“POCL will have to rely on the [Benefits Agency] or the service provider for accurate information for settlement.”
That was a concern. So it seems as though what POCL would like is to control the end-to-end process; is that a fair –
Paul Rich: They would certainly need to be able to assure it and accurately because, of course, those processes and the information flows from the system would have not only told the relevant client, like the Benefits Agency whether or not their service had been provided and settled but also would have told our own accounts team that it had been in parallel.
I think the – I think I remember that both – you see it says “Copy to Ian Gair, Tim Brown, Kevin Corrigan/Byron for comment”, Tim Brown was one of the assistant finance directors at the time and he would have assured it and I think I remember that the chief auditor, Peter Dent, also had input in the one of the documents I’ve read.
Mr Blake: If we move on to February of that year, 1996, can we look at POL00028288, please. This is a meeting note from the Evaluation Board. We can see there that the chair is Bob Peaple and I think you’re listed as one of the attendees?
Paul Rich: Yes, as I said yesterday, yes.
Mr Blake: There are representatives, as you said yesterday, I think, from the Benefits Agency, as well, who attend that meeting?
Paul Rich: Yes, and the Social Security Agency for Northern Ireland and DITA is the Department – I think that’s the Department of Social Security’s IT people.
Mr Blake: Thank you. Yesterday we heard from John Roberts who received updates about the Evaluation Board but these are the actual Evaluation Board’s own minutes. Can we look at page 3 of these minutes, please, and, focusing in on the bottom half of the page, that’s 2.7 and 2.8. I’m going to read those. 2.7:
“With respect to the certificate for Pathway, the Evaluation Board expressed severe reservations about the number of significant risks against the Pathway proposal. It questioned whether the cumulative effect would not lead to potentially late delivery and/or operational problems, and as such was cause for Pathway to be excluded from the [Invitation to Tender]. Particular concerns were expressed about the card technology with shortcomings in the associated management of fraud and about the dependence on Escher as a small company subcontracted to Pathway.”
Who was Tony Johnson?
Paul Rich: He was in the – I can’t remember. I was thinking about this the other day. I can’t remember which organisation he came from but he was part of the demonstrator team within the assessment team, the one run by Andrew Stott.
Mr Blake: So:
“[Mr Johnson] explained that these issues had indeed been given much consideration by the Demonstrator team. It had concluded that the Pathway solution was not fundamentally flawed and that from a strictly technical view point the hurdle had been cleared.”
Paul Rich: Yes.
Mr Blake: “In its overall decision, the Evaluation Board might wish to take into account the cumulative effect of any contractual issues with those from the requirements area.”
So not fundamentally flawed, that’s a phrase that we heard yesterday as well. That’s not a ringing endorsement of the system, is it?
Paul Rich: No, it’s not a ringing endorsement, I agree with that, but it was a statement we relied upon at the Evaluation Board, noting the risks, among others, that are in paragraph 2.7. It was – and as I also explained yesterday, I think, that when this eventually went to MaPEC Major Projects Expenditure, for the main Post Office Board, the paper would have had to have group IT technical concurrence –
Mr Blake: Absolutely.
Paul Rich: – and they said it is acceptable even if it’s not ideal.
Mr Blake: In terms of the three potential options, Pathway was the least technically strong of the three?
Paul Rich: Yes, I think I know what you’re referring to because it said the evaluation came third in – I can’t remember how many, but each one cleared the hurdle and each one had their own issues, is the truth. So you had to apply an overall judgement about the evaluation not only on its technical merits, which had to be acceptable and know the risks you were going into in trying to manage those, but also the other aspects from the other issues around operational management, contractual and PFI compliance.
Mr Blake: Also highlighted in the section I’ve just read are concerns about the dependence on Escher as a small company –
Paul Rich: Yes.
Mr Blake: – and also concerns about the card technology at that stage.
Paul Rich: Yes. The card technology was I read – forgive me if I’ve got his name wrong – Mr Cipione’s technical – I was given access to his witness statement a couple of days ago. You have to remember what the technology was like at that point. You know, it was a very different world then. So ICL was relying on magnetic stripe technology for this.
We looked at and considered something that came up from them, which was a card that included an integrated circuit, known these days more as a smartcard and, at that time in this country, that was very new technology, more expensive, quite risky of itself, gave Benefits Agency some concerns about the security aspects associated with having those cards for benefit encashment. So we decided that would be more risky.
As for the dependence on Escher, yes, I understand that as well. We knew that they had a proprietary bit of software called Riposte and it hadn’t been proven at scale because the demonstrator was, I think, an office in Ireland, basically, if I remember rightly.
But then –
Mr Blake: I think it may have been in the United States.
Paul Rich: Okay, I thought they went to see something that they trialled in An Post but I may misremember, forgive me.
Mr Blake: No, no, I think you may be right.
Paul Rich: I think they are an American company, I think.
Mr Blake: Absolutely.
Paul Rich: We knew that risk and we weren’t shy about telling ICL Pathway about that risk and they would have to address it if they were going to go forward and, hence, why we put the mitigating stuff around the risk later on.
Mr Blake: Can we look at 2.10, so that’s over the page. Derek Selwood: who is Derek Selwood, sorry? Are you able to assist?
Paul Rich: He’s, again, part of the assessment team.
Mr Blake: “Derek Selwood confirmed that risks and issues against service providers would be taken into account in the evaluation and selection process. The risks against Pathway would incur a substantial cost penalty given their number and severity. The issues identified during the Demonstrator would affect the Value Factor assessment, and it would be for the Evaluation Board to give due weight to that in reaching its decision.”
Now, sometimes a proposal might just make the cut but everybody in the room knows that, ultimately, the penalties will mean that it’s ruled out. How did you feel at that time? Did you have any feelings about Pathway?
Paul Rich: I knew there were some risks with it, as there were with others. It was very difficult, really, at that time, to go out and say, “Oh, look, here’s this system somewhere else that’s been done before”, because there was nothing like it and there was certainly nothing like it done under a private finance initiative. So we were all, I think it’s fair to say, learning because you couldn’t say – IBM couldn’t show us – or Cardlink couldn’t show us “Here’s something we’ve done extensively like this in the States and also transferred the risk of a large part of fraud at the same time”.
So you had to do that but – I don’t think it was with reluctance that we chose them. We were looking to try to secure an automation provider for both us and for the Government.
Mr Blake: While we’re on this document, can we just look at paragraph 2.12 below.
Paul Rich: Yes.
Mr Blake: I don’t need to read out the paragraph there but it suggests there was some confusion over the requirements for EPOS at that stage. Do you remember that?
Paul Rich: I’m trying to remember, sorry. I’m just reading the material. (Pause)
I think this was about when and where EPOS would be down the track, in terms of its releases. As I said, we put in the requirements that it had to be – the solution had to be able to replicate existing processes but we knew that it would need to migrate to better automated processes later on. We were trying to give priority to the DSS service, the benefit encashment service.
Mr Blake: Absolutely. That’s the very point that I make, that at that stage the non-Benefits Agency part was very much in its early thinking when it comes to something like –
Paul Rich: I think we’d fleshed it out but we also realised that it might change over time, depending on what other clients might want to be taken on, for example. There were competitive pressures from competitors, which the Federation of SubPostmasters weren’t slow to tell us about, from people who were – later on, you’ve seen it in other documents, were putting out – a competitor was putting out an automated payment terminal for bill payments, which the subpostmasters were waiting for Horizon to do.
Mr Blake: But EPOS itself was pretty fundamental to the non-Benefits Agency part of Horizon?
Paul Rich: Yes, and the join-up with the back office systems.
Mr Blake: It seems that in early 1996, at least, it was only very much in its early stages and there was, according to this minute, some confusion over what was required. Do you agree with that?
Paul Rich: I don’t remember it, is the truth, but that’s what it says here so I take it as read.
Mr Blake: Can we move on to a board paper. It’s POL00031237. In fact, this document starts as a board paper but can we go to page 9.
Paul Rich: Can you just tell me what date that is, please?
Mr Blake: The document on page 9 will assist, actually.
Paul Rich: Thank you. Oh, yes I remember this now. Thank you.
Mr Blake: I’m just looking for the date. The date of the –
Paul Rich: No, I understand. This (unclear: simultaneous speakers) the Evaluation Board more or less immediately while – once the recommendation had been made and mandates were being sought.
Mr Blake: So this is a minute from Bob Peaple to the Project Steering Committee?
Paul Rich: Yes. Yes, we knew it as the Joint Steering Committee, so I’m not quite sure about that but fine, okay.
Mr Blake: Paragraph 1 sets out the “Purpose”. So:
“The purpose of this minute is to inform you of the substance of the meeting of the Evaluation Board which I chaired yesterday to consider the report of the evaluation team on the re-tenders submitted by Tom, Dick and Harry.”
We went over this yesterday with Mr Roberts. I think you heard Mr Roberts’ evidence?
Paul Rich: Some of it, not all of it.
Mr Blake: So Harry was Cardlink, Tom was IBM and Dick was Pathway. Can we look at paragraph 10, please.
Paul Rich: Peter Mathison is the chief executive of the Benefits Agency, by the way, he’s not a Post Office person.
Mr Blake: Thank you. I don’t need to read paragraph 10 but essentially there were some wrinkles and they needed to issue an invitation to re-tender on 16 April; do you remember that?
Paul Rich: I can remember there was some iteration. I don’t remember the detail I’m afraid.
Mr Blake: That’s fine. Can we look at paragraph 13, which is over the page and could we focus in on that paragraph, please.
That addresses the technical aspects and, as you said earlier, on a technical level, all three satisfied the minimum levels; is that right?
Paul Rich: Yes.
Mr Blake: Can we look at paragraph 16, that’s over the page. It was ICL that was closest to the risk transfer sought; do you remember that?
Paul Rich: ICL, as far as I can remember, were the only ones that met that hurdle. I think Cardlink was eliminated on other factors and then IBM were only prepared to take risk at a pound for pound, and I think – I may have these numbers wrong, so forgive me, but it will be right in the order of magnitude. I think Pathway were prepared to put up about 200 million of risk transfer.
Mr Blake: You’re absolutely right. If we go to paragraph 30 to 32 on page 15, please. Would it be possible to highlight this? Thank you very much.
Essentially, Pathway (Dick) was closest to what sponsors had sought to obtain, particularly in terms of the risk transfer; is that right?
Paul Rich: Sorry, that was a question? Yes, yes, it was. Sorry.
Mr Blake: Paragraph 33:
“The Board recognised that an award to Dick would imply a need for a proactive management stance by sponsors.”
What did you understand by that?
Paul Rich: Well, any set of risks needs managing and it means that both sponsors had the need to put forward a suitable method of managing those risks. The way we decided to do that was, as I said yesterday, to create this programme delivery authority, jointly staffed by BA and POCL people in the same building, working together – which was an interesting cultural dynamic – headed up by a DSS person to give the client reassurance and because they had skills, supplementing that with external contractors who had technical skills where necessary and then create – I don’t know what – BA and DSS had their own project team, I think, also because I used to meet my equivalent, as it were, on what became the PDA board, but we also set up teams within Post Office in parallel to check what the PDA was doing and if there were any commercial or contractual issues.
I say that about the commercial contractual issues because, clearly, the contract hadn’t been let then and there was still this to come and then we had to have – we had to finalise our agreements with Benefits Agency on the back of that because that was a back-to-back arrangement. But we also had to cope with the nature of the private finance initiative in that, in what proactive management meant.
Mr Blake: If you are looking for who is responsible for taking forward that proactive management stance, who was that then? Was that the programme delivery authority or was it a particular individual?
Paul Rich: Well, the programme delivery authority on behalf of the two sponsors, and that reported to a Project Steering Committee, confusingly here, that’s why I differentiated before, which consisted of the chief executive of ICL, the managing director of POCL, and the chief executive, I think, of the Benefits Agency being supplied with information. So if there were issues to resolve, that programme steering committee was meant to resolve them.
Mr Blake: So there are lots of committees.
Paul Rich: There are.
Mr Blake: We’ve heard about lots of committees. Who within the Post Office would have been responsible for taking forward the proactive management stance?
Paul Rich: Well, the managing director essentially would have been mandated by the board. He delegated quite a lot of that to me to do things. He would have also taken a proactive stance, as I did, with the person running the programme delivery authority.
Mr Blake: So that’s Stuart Sweetman as managing director –
Paul Rich: He was by then, yes, I think.
Mr Blake: – yourself, and then somebody within –
Paul Rich: Peter Crahan was the programme director, having taken over from Andrew Stott, I can’t remember the exact date. He was another DSS person. George McCorkell in the Benefits Agency, and the PDA board was chaired by – ably, I may say – Alec Wylie, who was the chief executive of Social Security Agency for Northern Ireland.
Mr Blake: Can we go back to the document that we were just on but look at page 1, which is the note for the board.
Paragraph 11 on the second page, I won’t read it but it sets out the financial evaluation – 11 and below – and then can we go to the next page, and paragraph 14 is the “Non-financial evaluation”. One of the headings there is “non-financial characteristics” and it says:
“this reviewed suppliers’ performance against a number of characteristics, including customer acceptability, reliability and support, managing capability, etc. All three suppliers exceeded the acceptable level with the differences between them not significant for the purpose of discrimination.”
I think that’s consistent with what you told us earlier about the three?
Paul Rich: Yes.
Mr Blake: There isn’t mention there, at least, of concerns about, for example, the dependence on Escher or the card technology issue?
Paul Rich: No, hadn’t they been mentioned earlier or in the note?
Mr Blake: In fact, it mentions it slightly further down, so paragraph 15 that we can see there, “The Programme Evaluation Board recommended Pathway”, and that was endorsed by the Joint Steering Committee.
Then it’s paragraph 18, I think, that is the part of this particular note for the board that addresses the other technical risks in a little bit more detail. Could we have a look at paragraph 18?
Paul Rich: Yes, I see, okay.
Mr Blake: “Some technical risks were identified with all suppliers, and in some areas, Dick was considered to have higher technical risks that Tom and Harry. However these risks are manageable through;
“a strong technical assurance function, with support from the Post Office IT Directorate,
“rigorous testing at development, trial and roll-out stages,
“ensuring supplier contingency plans,
“a proactive technical management plan.”
Is that the proactive management stance that we talked about earlier that Bob Peaple was referring to?
Paul Rich: Essentially. I think I just fleshed it out a little bit more in what we actually did, rather than what we thought was needed to do, as it were, and it also, depended on the strength of the people we both put in to that programme delivery authority when we staffed it.
But I think for the last, certainly, points 2 and 3, I think that’s evidenced in all the bundles I’ve read, so far.
Mr Blake: So the Post Office IT Directorate were going to take responsibility for some of the technical assurance function?
Paul Rich: We would have placed people from within the Post Office IT function either from within Post Office Counters or from group IT into the PDA and, at the same time, they would have been assuring the board themselves through the group IT director or to whom he reported at board level, executive board level that –
Mr Blake: Can I just pause you there?
I am being told that the transcript has stopped. We may need to pause for a minute or two just while that is rectified.
Perhaps we can take a short break. I know it’s very early already but, seeing as this might take a few minutes, that could be our mid-morning break because we have a lot to do today.
Sir Wyn Williams: Whatever suits best, Mr Blake.
Mr Blake: Can we have a ten-minute break now?
Sir Wyn Williams: Yes.
Mr Blake: Thank you very much.
(10.32 am)
(A short break)
(10.41 am)
Mr Blake: Thank you, Chair, we can see you now.
I won’t go back to that document but I think the bottom line, from what we just heard this morning is that Pathway got through the process but it was very clear that they needed close management; is that a fair summary?
Paul Rich: As far as the – as within the remit of the PFI and I think what became clear later on and there are documents in the bundle that their idea of what PFI was slightly different to what the sponsors was. I think there’s a letter from Keith Todd later on in March ‘98 to Peter Mathison, chief executive of Benefits Agency, that spells out what they believed PFI meant and the level of unnecessary interference, therefore.
Mr Blake: Can you summarise that very briefly then. Was it your position that you could have more involvement and their position that you shouldn’t have the level of involvement that you were seeking?
Paul Rich: Yes. Yes, if you want to summarise that briefly, “yes” is the right answer. I think it was really around the extent to which they could, under the PFI – private sector are innovative and can bring their skills to the party – could take the business processes and we would have to rely on the outputs rather than the details of the design architecture underneath it.
Mr Blake: Can I just take you to your witness statement. It’s WITN04030100 and it’s page 16 that I’d like to look at?
Paul Rich: This is my witness statement?
Mr Blake: Yes.
Paul Rich: What paragraph?
Mr Blake: It’s paragraph 47. You say there:
“Some post offices were not easily suitable for IT infrastructure, as they did not have the necessary space or equipment. Moreover, Pathway underestimated the amount of new software development needed from its subcontractors, eg Escher. A further issue was that the BA/DSS [CAPS] which was to feed data to Pathway’s card systems, was not ready …”
That sounds very much like the problems that were raised at the Evaluation Board pre-Invitation to Tender that we saw first thing this morning, don’t they?
Paul Rich: Some of them. One of the three but not the other two.
Mr Blake: There’s the issue with Escher and also the card system – concerns about the card system.
Paul Rich: No, I don’t think that says – I’m sorry, Mr Blake, I don’t think that says concerns about the –
Mr Blake: It’s the next page, sorry.
Paul Rich: Ah yes, I see. Thank you. Sorry.
Mr Blake: That paragraph begins by talking about the infrastructure. Keith Todd may, in due course, say that Post Office couldn’t reasonably have believed that their premises were fit for automation and that ICL weren’t given a proper opportunity to inspect. Would you agree with that?
Paul Rich: I said it was a learning process and, certainly, one of the things we knew, but we learnt more systematically, if I can use that word, was that of our 19,600 post offices they’re not a heterogeneous set. You know, you had Crown offices with 20 counter positions and a front room in the Orkneys. I think what Pathway underestimated, reasonably, was the amount of the network that wasn’t covered by ISDN. For those of a certain age, that’s a sort of predecessor to broadband, I suppose, on which their system relied for messaging and, therefore – and typically in the smallest offices, of course, or in the most rural ones where BT, at that time, or Energis hadn’t done.
So they didn’t know that until they surveyed and we wouldn’t have been able to tell them that in this statement of service requirement and, of course, the ergonomics as well about counter space and, hence, why we always insisted that both customer acceptability and how long transactions would take – in case it affected our quality of service, queueing time, for example, in bigger offices, and user acceptability, in terms of agents or staff being able to use this, both physically and in terms of software – were always included.
I think that’s probably why, when we first realised we wouldn’t be able to achieve post contract what we set out to, there was something called a no-fault re-plan because the other thing that was mentioned in there – I don’t know if you’re going to ask me about that – was the DSS issue.
Mr Blake: Yes, I’ll come to the no-fault re-plan shortly. You also highlight in your statement that Pathway had underestimated the training time required.
Paul Rich: That’s true.
Mr Blake: You said, in terms of infrastructure, their underestimation may have been reasonable. Was it reasonable in respect of training time?
Paul Rich: We thought they should have known that it would take – as Mr Cipione said, we were acquiring a system not a piece of software and that system is about people and the human interface with that. That was always at the heart of our business, unlikely as it may sound now.
But the training that they set out, which they did – which they were contracted for, for part of that system, which they subcontracted to a company called Peritas, from memory, probably wasn’t as extensive enough and, certainly, we pushed back hard at that once we had that pointed out to us, engaging with the Federation and with subpostmasters and – et cetera.
Mr Blake: Is that something that thought was given to during the earlier stages: the evaluation stage, the Invitation to Tender stage?
Paul Rich: I think – did we give it enough thought? That’s a good question. Certainly, they had to demonstrate capability. You know, it wasn’t like going to a Microsoft and saying “I’ll have that computer”. It was buying a whole system with all that went with it, including putting it in because we were an unautomated business of 19,600 offices with 70,000 people involved and it doesn’t take a genius to work out that that culture change is going to happen overnight without professional training.
I’ve read other documents later on, after my time, when roll-out began, where I’ve only seen those since you’ve sent me – the Inquiry sent me the bundles as I wouldn’t have been there – I’ve seen that, even in late ‘99, one of our senior managers was writing to Pathway saying “The training is still inadequate”.
Mr Blake: Moving to February ‘97 to spring ‘98, as a member of the PDA board, you received testing reports and something called technical concurrencies. Can you briefly describe what technical concurrencies were?
Paul Rich: Can you show me had – they mean different things at different contexts, that’s all.
Mr Blake: I only need a very brief explanation.
Paul Rich: Technical concurrencies meant is this stuff fit for purpose and is it working as we think it should or it is likely to work as we think it should and is it capable of working as we think it should? Certainly, as I’ve described before, the Post Office Board was relying on technical concurrence from our own people, from the group IT people downwards, in order to assure it’s approval of the business case.
Mr Blake: You, explained in your statement at paragraph 54 and 55 that there were delays in spring ‘97 and into 1998. One of those reasons was an underestimation by Pathway of the effort and time needed to develop the services and the other was an issue with Escher.
Paul Rich: Yes.
Mr Blake: The Escher point again, something that we’ve heard a lot about this morning, mentioned that pre-ITT stage.
Paul Rich: Yes. It becomes a running sore is the truth. I know there were – it wasn’t like ICL were ignoring it. You know, they were trying to fix it and they were putting more and more people, technical people, onto it.
I think the National Audit Office report says this as well, as well as PA, that ICL thought there’d be more systems integration than systems development involved. You know, they thought they could take pieces of the jigsaw and slot them in but when they tried to slot them in and then tried to scale them up, it needed systems development as well as integration skills, particularly around complex business rules like the DSS.
Mr Blake: In terms of Pathway’s underestimation of the effort and time that was needed, was this proactive management stance implemented at that stage? Do you think that ICL were being effectively managed?
Paul Rich: I think we were scrutinising them and I think if we hadn’t have done we would have been tempted to say, “Okay, that’s all right, go ahead then” and we never did that. Personally – and you have got evidence in the bundles, I would never compromise quality for speed, and I think I can – we can demonstrate that by the number of replans, both in terms of roll-out to offices and to what was in the various releases that there were following the Benefits Agency one, where we kept staggering them and simplifying them to give it its best chance.
But we wanted to try to fix before rolling out and that was why the approach was taken about an initial Go Live of one office and then of ten offices and then a 200-type roll-out before any acceptance of the system or roll-out, in order to be able to test because some of the things you can’t test unless it operates.
You can test some things, model office testing and end testing, as it’s called, or user acceptability testing, all three of those, but until you actually put it in action in a safe environment, if I can put it that way, so you can tell with real customers and real agents using it, you need to understand what the result of that was.
Mr Blake: You talk in your statement – it’s paragraph 57 – about the causes of delay and roll-out being the same as the causes of the no-fault re-plan.
Paul Rich: Yes.
Mr Blake: Very briefly, what were those causes, so far as you saw them?
Paul Rich: Inability to demonstrate to us completely – to the Post Office, that it was able to work well in offices before rolling it out, an inability, from the DSS point of view, to securely do the benefit encashment service as far as they were concerned. That was their end testing.
Two other points, just in support. Complexities around the installation of the network and the ergonomics that I mentioned earlier but ICL were more aware of that by then, so by the time after the ‘96 re-fault plan, when the new master plan was – there’s a document that describes it in March ‘97, which was done for all parties as a re-plan, and then there was disappointment after that because ICL failed to deliver again to time.
But there was also the DSS issue of whether their own systems were fit for purpose and had been progressing as quickly as ICL believed to feed the system. I think Keith Todd, in a letter, called – said to Peter Mathison said that he believed DSS had “misrepresented” the state of the CAP systems.
Mr Blake: You talk about disappointment. I’m going to move to the summer of 1997. Can we look at POL00039669. Now, this is a new document, in the sense that it has only very recently been shown to you, so if you need more time to look at this document, please do let me know.
Paul Rich: Can I have a look? It hasn’t come up on the screen.
Mr Blake: You are going to need more time than that!
It should be familiar now and it’s only one sentence that I’d like to take you to.
Paul Rich: You gave me this at 2.00 yesterday.
Mr Blake: Indeed.
Paul Rich: Sir, I haven’t fully reflected on it, I have to tell you, Chair, I’m sorry about that, but I read it overnight.
Mr Blake: It’s the part just below “On Pathway” that I wanted to draw your attention to. This is a letter to you from Stuart Sweetman.
Paul Rich: Yes, it’s an internal memo because I was going on holiday, yes.
Mr Blake: It says:
“The technical release slippage is a cause of major concern for the [Benefits Agency], not so much for its direct impact but because it is a ‘very poor’ signal of ICL’s capability, so soon after a re-plan.”
Am I right to say it sounds as though you are quite concerned by that stage?
Paul Rich: I’m concerned and certainly DSS are. I think this is the time at which we asked – I think you will see in Peter Copping, later on today – we asked PA to come in and do an independent review of where we were to see – in mid-‘97, to check because, you know, one slippage on a no-fault re-plan, soon after contract letting, as we are all learning is one thing but then for ICL not to be able to do what they said they would do at that point three months later is another.
Mr Blake: Why would you raise it with Stuart Sweetman?
Paul Rich: Because Stuart needed to know because he was part of the programme steering committee. You know, I said earlier that major issues would have had to be resolved and I would have always kept the managing director, to whom I was a direct report, appraised of the situation because he may well have been dealing with other stakeholders like politicians, ministers and people like John Roberts, actually, as well.
Mr Blake: I’m going to take you to one more document of the same period. This is 6 May, so a few days later. It’s POL00039668. I apologise, this is, again, another one of those documents that you have only recently seen.
Paul Rich: It’s not on my screen, sorry, Mr Blake. Ah yes.
Mr Blake: It’s from you to Min Burdett. Who’s Min Burdett?
Paul Rich: Min Burdett is someone within the technical assurance team. She’s a technician working to Mena Rego – you see Mena there and Mena was direct report to me, not as part of the PDA but within Post Office Counters. Mena had obviously asked her to have a look at the latest PDA documents that Pathway had put forward as part of their proposal to re-plan.
Mr Blake: Can we look at the penultimate paragraph on that page, please. It says there:
“The main concern that POCL has is that it is not clear whether the right tests are in place (either as part of DIT or in the subsequent Model Office)to ensure the integrity of the new Benefit accounting, settlement and reconciliation process.”
Paul Rich: Yes.
Mr Blake: Settlement and reconciliation process, obviously, we know is quite important, quite fundamental, in terms of the work of the subpostmaster.
Paul Rich: Yes, and I’m pleased that we were doing the work to try to assure that and absolutely check it out.
Mr Blake: Were you concerned about it at that stage?
Paul Rich: I was concerned – I would have been concerned about all aspects of the programme that hadn’t been running to time.
Mr Blake: Can we go to page 3 –
Paul Rich: I think in paragraph 2 above it shows you that we were prepared to move some – we weren’t prepared, sorry, to accept some of the planning – squeezing in of new functionality without it being properly tested first.
Mr Blake: Paragraph 3, the first bullet point, it says:
“Assuring that the new Accounting, Settlement and Reconciliation end-to-end process works. This is a POCL ‘must have’ and yet is not clear how POCL is going to know before the go-ahead is given that the new process will work.”
Again, that must have been quite a concern at that stage.
Paul Rich: It was a concern and if you note the timing of this, which is –
Mr Blake: It’s 6 May 1997.
Paul Rich: If you note the timing of this, there was a PDA report which I thought was – sorry a PA report, which I thought was fair game, but one of the documents you gave me yesterday showed my reaction to a draft which said it had some gaps. But it was fair game for much of it. Then it also was about the time – I know you questioned – I saw that you questioned John Roberts yesterday about the lessons learnt exercise he had asked, which was July, I think, so two months after this, and, you know, I gave him that, as he called it, I think, a warts and all.
Mr Blake: Absolutely. Shall we turn to that? That’s POL00028953.
Paul Rich: We haven’t scripted this, have we? So that’s good.
Mr Blake: This is your covering letter.
Paul Rich: This covering letter to my colleagues on the Counter Executive Committee because it was important that they knew about it because it involved actions across the team.
Mr Blake: Can we look at page 7, please. Can we look at the left-hand side of page 7.
Sorry, it’s the page before that. Thank you. Perfect.
This is a passage that I took John Roberts to yesterday, “there is a need to review enabling organisations”, and it says further down there concerns about the end-to-end.
No, sorry, over the page, sorry. Sorry, it’s the page before then. It’s internal page 5. It’s certainly page 7 of my document.
Paul Rich: Is this the – what’s the “Key Improvement Lessons” it’s under –
Mr Blake: “We need a better idea of connections to our other new systems”, I think this is something you spoke about earlier.
Paul Rich: Yes.
Mr Blake: Is the concern that’s being raised there that there weren’t proper systems in place for POCL to understand why Horizon might not be working properly?
Paul Rich: I think that what became clear in this candid report from me was that I said it’s a learning exercise for us in becoming an automated organisation. I think I say that somewhere in here, and one of those things was that we had a number of projects in train that weren’t to do with Horizon, either existing systems or back office systems, that were going through their own change and what we didn’t have in one place was something to join those all up and plug Horizon into it, if I can put it simply like that.
So the – sadly, I can see that most of the actions fall to me.
Mr Blake: Absolutely. Can we look at the top document that’s shown on screen and it’s the bottom two bullet points on the right-hand side.
Paul Rich: Yes.
Mr Blake: “end-to-end testing procedures will need to be transferred from the PDA, and supplemented as release planning migrates back to POCL after the system is accepted
“a process for live trial acceptance is being devised to ensure collective ownership across POCL functions.”
So that’s your name on the right-hand side.
Paul Rich: Yes. I mean, it is also “after the system is accepted” on the penultimate point which, of course, it hadn’t been and wasn’t until late ‘99.
Mr Blake: Is that penultimate point saying that, once the PDA has finished its work, that end-to-end testing will need to be assured at the Post Office end?
Paul Rich: Yes, for the Post Office systems, not for the DSS systems –
Mr Blake: Yes.
Paul Rich: – which is the other part of the end-to-end for some services, obviously.
Mr Blake: The final bullet point, live trial acceptance, the processes were going to be Post Office processes.
Paul Rich: Yes, and, as it says there, “to ensure collective ownership across POCL functions”.
Mr Blake: Now, your name is on the right-hand side.
Paul Rich: Yes.
Mr Blake: Does that mean that you were responsible for taking those forward?
Paul Rich: I was responsible at – you see it says “Paul (ATSG)”?
Mr Blake: Yes.
Paul Rich: That’s the automation transformation steering group, another fantastic name. I think it became known as harnessing technology in post offices later on but it was – basically I was asked by Stuart to set that up and bring together the project managers of these various technical projects, to be able to talk to each other and create a milestoned plan that ensured they joined up. I mean, at this time we were in the early stages of year 2000, for example.
Mr Blake: So as things went forward, who would have been taking that forward?
Paul Rich: After I’d left in early ‘99, that would have fallen within the remit, I imagine – I can’t be certain about this, you’d need to ask – of Dave Miller, I would have thought.
Mr Blake: Can we –
Paul Rich: Dave Smith was the programme manager, the guy who apparently got up that interesting slide yesterday.
Mr Blake: Can we go on one page after the bottom page here. So it’s internal 7 but it’s my page 9 and we have there, on the left-hand corner – this is again something I raised with Mr Roberts – robustness is being raised as an issue there.
On the right-hand side:
“Collective nerve needed to ensure no compromised on quality for sake of speed and to retain programme focus.”
That’s something that you have told us about already today.
Paul Rich: Yes.
Mr Blake: Was there some pressure then? Why would this need to be said?
Paul Rich: Because I think – remember I’m talking both to – this is a report, essentially, for the John Roberts Counter Automation Steering Group and I’ve copied it to my Counter Executive Committee colleagues and, therefore, all the main players within the Post Office or within Post Office Counters are recipients of this and I just wanted us to be clear between us that we remained committed to not being expedient for the sake of speed, given the delays.
Mr Blake: Were there some who just wanted it to happen quickly?
Paul Rich: No, I don’t think so. Not within the Post Office, no. But in managing the disappointment of the slippages within Post Office and Post Office Counters, there were other stakeholders to manage as well, including Government ministers who’d have been shown timetables before. You referred me yesterday to the Benefits Agency POCL memorandum of understanding, which pre-dated the awarding of the contract and, within that, there was an ambition to have completed the roll-out, as far as DSS were concerned, by the end of ‘99 and we were nowhere near that.
Mr Blake: So were the external pressures to speed up?
Paul Rich: Or to take a different approach. I mean, I think I said elsewhere, and it may be at the bottom, I can’t remember, at the bottom of this document.
Mr Blake: Perhaps we can look at internal page 8. It’s page 10 here.
Paul Rich: “DSS – political issues”, and as my note to Stuart – as I was flying off to Germany that day – said, after I’d met George McCorkell for dinner the night, my Benefit Agency equivalent, I think the slippage after the re-plan had dented confidence within DSS, and you have to – you asked me about our relationship with the Benefits Agency yesterday. Probably more DSS than BA, not quite the same thing now. Different culture than us and much more command and control, much less open.
You can see from the style of this document this is a very open document. It underpins our – we had something called Business Excellence, Total Quality culture where no denial was a characteristic and putting the customer first was a characteristic at that time. DSS wouldn’t have thought like that.
They always, I think, had the ACT option in the background and their systems, which not me but the NAO said weren’t ready for purpose at the beginning of the – were getting more fit for purpose.
Mr Blake: The threat of termination that’s mentioned on that document, though, that must have had some impact on getting the job done quickly?
Paul Rich: Well, clearly we agreed. There’s documents that we agreed with the Benefits Agency through the programme delivery authority and its lawyers to put a breach notice in November ‘97, I think, from memory, so after this, but also committed to keep working on the programme while that was put forward.
You heard John yesterday say we had a difference of opinion about whether to follow that up later on with a cure notice, a 13-week cure notice. There will be lawyers here that know better than me what a cure notice is, I’m afraid, but I think it’s essentially putting them on notice that, unless they didn’t get – fund this in 13 weeks, there would be termination.
Mr Blake: During that summer there was still some significant technical problems. I’m going to take you to a document POL00028311. That’s a programme delivery authority board meeting, 21 August 1997. Can we go to page 4. That’s paragraph 2.1.4.6. It says there in the third sentence:
“POCL also had problems with testing especially Electronic Point of Sale System … Pathway reported that their testing strategy was under review and agreed to pay particular attention to EPOSS.”
Paul Rich: Good. I notice it is Mr Coombs direction and Mr Coombs was someone that Pathway brought in to strengthen their technical team. He was the ICL technical director at the time and I think he was parachuted in to Pathway.
Mr Blake: Can we go over the page to paragraph 2.3. There you highlight that more emphasis was being placed on live trial and quality rather than speed. So that’s again the very point that you made in the earlier document. Again, there seems to be a looming pressure coming from somewhere for speed. Would you agree with that?
Paul Rich: Not from us. I mean, you know, we had what we thought were a set of committed timetables after a re-plan in March ‘97 and they were slipping. What do you do about that, is the question, you know, and so we had another re-plan and we questioned severely from the documents, some of which you’ve just seen, about whether Pathway’s proposals to catch up in terms of release software were really credible. We wouldn’t have agreed to something that was intrinsically risky technically, as far as we were concerned, before it was bottomed out.
DSS, as I said, had promised their business case, as I understood it, belatedly, from documents I’ve seen and from the NAO report to remind myself – had predicated to the Treasury something like 15 million a month on fraud savings, as the basis of their business case, in order to – once it had been rolled out and once it was working. So, clearly, they had – I mean, the delays were hurting every party financially.
There was no winner here. ICL weren’t getting paid because the PFI structures. We were having our competitive position damaged. We were disappointing subpostmasters and our staff whom we said “This is coming”.
But so there wasn’t, you know, severe pressure to get this done but we had to, as I say, hold our collective nerve to ensure that what we put out was good enough.
Mr Blake: But there’s document after document saying “We need to emphasise quality rather than speed”, and, I just wondered, somebody must have been asking for speed rather than quality?
Paul Rich: As I said, I’m trying, to be honest, really honest and candid and to the best of my recollection. There was undoubtedly pressure on Benefits Agency from DSS and I would imagine the Treasury. I don’t know I’m speculating on that. They were pretty close, DSS and Treasury. They were a big-spending department so they would be.
You know, they would have been saying, “What’s going on here? Why can’t we do ACT instead?”
Mr Blake: You mention the PA Consulting report and that was in October 1997. I’m going to spend a little bit of time on that.
Paul Rich: I thought it was a bit earlier than that, I’m sorry.
Mr Blake: The report itself was October 1997 and that’s actually one of the questions that I’m going to begin with which is did you input into the report before it was published?
Paul Rich: I did, because Peter came round and saw most parties from within the – because he was commissioned by the PDA from Alec Wylie, so all parties involved. I think you reminded me, actually, from the document I saw last night – thank you – about my initial reaction to it was that one of my queries was I don’t think he talked to enough people within Post Office Counters to get a fully rounded view on people like who were involved in the automation steering or the finance people, although we asked him to do it. But you’d ask Peter about that yourself.
Mr Blake: You said at paragraph 63 of your witness statement that PA Consulting report of October 1997 found no fundamental technical issues with the system. That sounds a little bit like the comment we heard yesterday, “not fundamentally flawed”. Did you have lesser concerns?
Paul Rich: Well, it’s a load of lessons that were emerging and I tried to dispose in that July report for all parties. You know, I could only take – I could observe what was happening with other people and give my opinion on it. I could only action what I thought was necessary within Post Office or recommend action.
Mr Blake: Can we look at POL00090015.
Paul Rich: But you are right that we did rely on quite – that it was – it could work. It wasn’t technically intrinsically flawed.
Mr Blake: Absolutely. This is precisely a document that I’m going to take you to. Again this is one of those new documents –
Paul Rich: This is the one I was referring to.
Mr Blake: Yes.
Paul Rich: There you go.
Mr Blake: So if we look at that document, it’s the last page of that document and it’s a letter from you to Peter Copping.
Paul Rich: Yes.
Mr Blake: This is – it seems like an important point. So you’re saying there –
Paul Rich: What date is this please, Mr Blake?
Mr Blake: It’s 8 September 1997 so shortly before publication. Publication was in October.
Paul Rich: I see. Thank you.
Mr Blake: “Two points do occur: first, on the basic technical question of whether there’s a basic ICL Pathway design flaw or not … combined with the sheer scale of what we’re trying to do, makes the programme inherently unstable. The report is silent on this explicitly at the moment, though it implies the design is feasible. I wonder if people who worked on the initial technical evaluation (including, if I recall correctly, a PA consultant) could help [me] here?”
So this seems to be you drilling down on precisely the issue that we were addressing yesterday, that because you were focusing on “fundamentally flawed” –
Paul Rich: We wanted to know because, as I said, we had a culture – I don’t know what it’s like today in Post Office Limited – but we had a culture of “no denial”. We wanted to know. If you didn’t know, what could you do about it?
Mr Blake: “Instability” is perhaps – would you agree it would be a better test than “fundamentally flawed”, if you are agreeing a system?
Paul Rich: I mean, I think my language was a bit, you know, imprecise there. “Unstable” could mean a number of things, couldn’t it, and I can’t remember exactly what I meant there. I didn’t know if it meant “undoable” or “it might fall over”; it could mean either of those things.
Mr Blake: I think “fundamentally flawed” you would understand to be doesn’t really work, whereas –
Paul Rich: Yes, “fundamentally flawed” – it’s the “inherently” bit. You know, “inherently” implies that – or to me at any rate – that it will be difficult to fix and it might be something structural and systematic. If there are technical issues and flaws that can be addressed and tested, that’s a different matter.
Mr Blake: Would it be fair to say that you’re asking there, is it unreliable even if it works?
Paul Rich: I don’t think I had that in mind when I wrote it, is the truth.
Mr Blake: Were those questions that were being asked at the time?
Paul Rich: I’m trying to remember honestly. We certainly will have – “unreliable” will have been really manifested during a live trial if it was unreliable or not and we hadn’t gone into live trial other than a few offices at that stage. So it would have been difficult to know. CF my earlier answer about; some testing you can’t do until you actually – you can do all the testing – It’s like playing a football match; you can do all the training but you have to actually be in a match to see if it works.
Mr Blake: So it’s the acceptance and the testing live trial –
Paul Rich: Live trial had to come before acceptance. That was a – very fundamental stuff and the issues that came out of live trial – or around live trial in the other forms of testing – would have been logged and then addressed to Pathway and the seriousness of them, either something that make it unreliable or the instance of them, you know, the frequency of them, would have been – would have determined their seriousness and their prioritisation to get fixed before moving on, to my mind.
Mr Blake: Do you think you got an answer to that point on whether it was inherently unstable?
Paul Rich: I think the October one, report that came out, said it wasn’t fundamentally flawed and I took that as including in that.
Mr Blake: So your understanding of “fundamentally flawed” would include whether something –
Paul Rich: Whether it was reliable to work, you know, because if it was fundamentally flawed it wouldn’t.
Mr Blake: Let’s look at the report itself. That is at POL00028092. Can we look at page 7, please. So although, as you said, the finding is that the system isn’t fundamentally flawed there were at least some concerns both in relation to POCL and in relation to Pathway at that stage that were raised by PA Consulting; would you agree with that?
Paul Rich: Yes, we did. All three organisations got some lessons to learn.
Mr Blake: If we look at those three bullet points at the top –
Paul Rich: Yes.
Mr Blake: – and actually can we look at the top half of the page including the paragraph that begins, “our key concern”. So there are some concerns raised in those bullet points and then it goes on to say:
“Our key concern is that the skills required for many of the new senior posts are, in our opinion, not those we would have expected to find as part of POCL core competencies …”
So there are concerns there, it seems, about a lack of skills within the Post Office; is that right?
Paul Rich: Lack of those skills particularly true as it says to implementation, management and contract and service. I wouldn’t have agreed with him about contract management. I think contract management, he’s talking about a technical contract and how you manage that. Service management, I would have agreed because, I said earlier on, we were learning to be an automated organisation and an automated organisation needs a central service management function typically. We knew that: we were trying to address it. It was complicated by the fact that the PFI contract was let out, so some of that service management, as Mr Cipione points out, was contracted to Pathway to provide, in terms of helpdesk, support desk, system desk, incident log-in, all that stuff, but you still needed someone to be able to manage that provider.
Mr Blake: The third of those bullet points, they concern EPOSS:
“In the longer term there’s the issue of developing the POCL requirements for EPOSS and the supporting computer applications. We understand there is much still to do here, which will require additional resource.”
EPOSS – we heard about earlier this morning didn’t we – at an early stage there hadn’t been enough thinking about EPOSS and again in October 1997 there were concerns about there being much still to do in relation to EPOSS?
Paul Rich: I agree with that. There was still much to do and it’s not a defence or an excuse, it’s not what I’m here for but the prioritisation on the benefits service, which could have put a whole system in jeopardy, or the whole project in jeopardy, we seemed to think we should get that right before moving on. We understood it needed to be done and we did put extra resource on.
Mr Blake: Can we look at page 8 and it’s M3.4 on that page. The report then goes on to talk about concerns with Pathway and it’s over the page that I would like to look at and it’s the first paragraph:
“Much of our review at Pathway has focused on the robustness of the technical solution since this has become a significant concern for the sponsors and Pathway. Whilst we have been able to obtain satisfactory answers to all our questions, in particular regarding the way security requirements will be incorporated and on scalability and performance, there must continue to be reservations in all these areas until the final design is baselined and then realised. It is also important to note the dependency for technical success on Escher which is a small, Boston-based software house and the source of the Riposte messaging software which is at the heart of the system.”
Now, again that’s something we spoke about earlier that was –
Paul Rich: And he’s repeating what we knew that Escher – they were dependent on Escher and what were they doing about it to put it right. I’ve been trying to remember that actually. I honestly can’t recall what extra resources or what actions Fujitsu took or Pathway took to – I can’t remember if they brought Escher in or bought them out or got them in but they certainly put extra technical resource to manage them.
Mr Blake: Can we look at your statement which describes replans that took place around this time. It’s WITN04030100.
Paul Rich: Paragraph, please?
Mr Blake: Paragraph 68, page 24.
Paul Rich: Thank you.
Mr Blake: I think this is talking about this particular time:
“This was my understanding of the technical difficulties with Horizon at the time but I would also add that those Post Offices to whom the service had been rolled out largely reported that it was going ok. They did not raise many technical difficulties and indeed many post offices who did not yet have access to the system were saying that they wanted it.”
Now, a reading of that paragraph might be that it contains quite a few caveats in it. At that stage it would own have been a small number of post offices who had Horizon installed I think less than 200.
Paul Rich: Yes, about 205-215.
Mr Blake: There was also limited functionality at that stage.
Paul Rich: Limited functionality, yes. Both of those are true. I tried to express that. Forgive me if I didn’t.
Mr Blake: If some subpostmasters were having some technical difficulties at that stage, when it was still a small project, would that have been a concern?
Paul Rich: Yes, but my memory of that is that the nature of the technical difficulties weren’t the ones that were being experienced, apparently, during roll-out, around the lack of balancing and that sort of stuff.
Mr Blake: So what was your understanding of –
Paul Rich: I think it was things like screens freezing occasionally. I might be wrong, there might be one or two. I can’t be certain but I do know that I can remember Dave – you will have, to ask Dave Miller – Dave Miller having a meeting with many of the subpostmasters who were in the trials or going round there and asking them the direct question – I think it’s in one of the Federation documents – saying “So given all this, would you want us to roll it out further?” With a resounding yes.
So that isn’t to say we were complacent. I wouldn’t want to give that impression. We weren’t. But the whole purpose of doing the live trial was, as I said earlier, was to try to get those experiences from those who were actually using the system and get some feedback into that.
Mr Blake: Could I ask for the witness statement to be brought back on screen on the same page if possible. Thank you very much. Can we look at paragraph 69. There you say:
“… the natural forum to raise these difficulties would have been at the meetings between the NFSP and the network director.”
Who was the network director? Was that Jonathan Evans?
Paul Rich: Yes, it was.
Mr Blake: So you would have expected problems to have been raised by the NFSP at that stage; is that right?
Paul Rich: The NFSP, as John said yesterday, certainly weren’t in the Post Office’s pocket. They were vociferous if they thought there was an issue and if there had been substantive – well I imagine, I’m speculating – I imagine if there had been substantive complaints by those subpostmasters who had used the system because of these sort of issues. I’m sure they would have been alerted to it and they would have been not slow in coming forward with those.
Mr Blake: But the number of post offices at that stage was very small.
Paul Rich: As I was saying, yes.
Mr Blake: What did you see as the role of the NFSP in that regard then?
Paul Rich: Clearly they – they were with us bringing together subpostmasters to get their reaction. I think the meeting I alluded to before, which I read about when Dave Miller went, I think there was an NFSP executive member there for that. The NFSP have been involved early on in the genesis of the Horizon project against threat of ACT and therefore the threat to the national network. They clearly would have been representing their members there, fiercely, to ministers.
Mr Blake: Do you think the NFSP were given much of a role at that stage?
Paul Rich: Given?
Mr Blake: Well, did they have much of a role, much of a say, in what was going on?
Paul Rich: They were kept informed. They went to meetings with Pathway, I think, during the bidding process. I think all three bidders set out stalls at a National Federation of SubPostmasters conference to show these sort of things. We hadn’t gone and we didn’t think about that until I was in the process of setting up Post Office Counters Limited we didn’t think about putting them on the board, for example, or giving them a non-exec role. I think that would have been a step too far for us at that point.
Mr Blake: Can we look at POL00028137, please. Now, this is a very early meeting, so some years before the period that I’m talking about?
Paul Rich: Yes, this is in the very genesis. This is just before – just to give context to this, the MOU you showed me yesterday which was ‘95 there were earlier non-legally – non-legal versions of that to which this was a contributor.
Mr Blake: Can we look at the penultimate paragraph of that and the final sentence of that penultimate paragraph:
“NFSP involvement would be kept to a minimum.”
Do you think that there was an attempt to keep NFSP involvement to a minimum?
Paul Rich: No. I don’t. I’m surprised about that. Did I write those notes?
Mr Blake: That’s a minute of the –
Paul Rich: Yeah, I wonder who wrote them. For example, Andrew – no, that’s not true. I was going to say he was the guy who dealt with the Fed but probably wasn’t at that point. No, I don’t – we did engage with the Federation around those issues but there were boundaries. You know, there had to be boundaries. They were a trade organisation representing people and we took them seriously because they were our people too.
Mr Blake: Do you think some people took a view that they shouldn’t be getting involved in the technical side of things?
Paul Rich: Well, I don’t think they had technical capability but certainly we were keen to include subpostmasters and tell the Federation about the user testing.
Mr Blake: I’m going to move on to November 1997 and that’s a month after the PA Consulting report. Can we look at POL00028599. This is an interim business continuity status report for the period 20 to 26 November 1997 and you were a recipient of this report.
Paul Rich: Was this one of the new ones, Mr Blake, or one of the old ones?
Mr Blake: I believe it’s an old one.
Paul Rich: Okay, fair enough.
Mr Blake: I’m only going to take you to a paragraph of it. Can we look at page 2, paragraph 1, halfway down that paragraph it says:
“The primary purpose is to identify any issues (actual or potential) that might give rise to considering a suspension of the Congo 4 roll-out or regression from ICL Pathway services.”
Do you remember the Congo 4 roll-out?
Paul Rich: I will probably get this wrong. My memory is failing me now from however long ago. Congo – I can’t – I wouldn’t be able to give you the detail of what Congo 4 roll-out. There was Congo 4, Congo 5, Congo 4 plus, we knew it also as 1C which I think it said before but I think these were releases relating to other functionality other than the benefit encashment service.
Mr Blake: At the bottom of that page it makes clear that there are no matters that need escalating for consideration of a suspending of the Congo 4 roll-out but it’s paragraph 3.2.1 that I’d like you to look at and that’s on page 3. Can we possibly blow up that paragraph. Thank you very much.
“After assurances following a previous duplicate payment … a new duplicate payment situation has occurred at Bath Road SPSO. This was due to the Post Office not being able to ‘poll’ for 8 or 9 days, therefore the system was unable to identify that the original payment had already been made.”
Now, we heard from Mr Roberts yesterday that he wasn’t really involved – or his evidence was that he wasn’t significantly involved in this kind of level of detail, individual branch level of detail – but this is something that would have been brought to your attention at the time, that level of detail?
Paul Rich: What date was this, please?
Mr Blake: It is November 1997.
Paul Rich: ‘97, okay. Yes, it would have been then, yes.
Mr Blake: What do you understand by that paragraph?
Paul Rich: I understand that there seemed to be an issue, as it says, not being able to “poll” that ICL would need to fix.
Mr Blake: Can we go over the page please to 3.4.1. Can we look at 3.4.1 – thank you very much:
“One of the new PMSR reports introduced at the beginning of release 1c, does not appear to be working correctly. It did not pick up the Bath Road duplicate payment and report it as an unmatched encashment. Pathway have stated that they will seek an urgent fix to this.”
Can we look at the paragraph below the two paragraphs below:
“Another concern is that one of the transactions involved in this incident didn’t come through to ABED and wasn’t reported on the CBoS report. Pathway aim to address this as part of the fix being applied to the above.
“This has a double impact on transaction processing:
“[first] it creates an error against cash account.
“[Second] settlement with BA is based on an incorrect sum – Pathway are looking to manually amend the CBoS report as an interim measure. This has an impact on POCL accounting.”
Pausing there, do you understand that second bullet?
Paul Rich: I don’t know what a CBoS report is, honestly I don’t.
Mr Blake: Were you aware that Pathway was able to manually amend certain things in branch – in the branch accounts?
Paul Rich: Well, from this – not that I remember – but from this, it’s in black and white, so I must have read it at the time. But I don’t remember that and I don’t – and it would have been with Pathway and the PDA to fix.
Mr Blake: It says, “This has an impact on POCL accounting”; do you know what that meant there?
Paul Rich: Well, presumably the accounting would have been – had to have been adjusted to correct the error.
Mr Blake: So if there’s a manual amendment it would have an impact on Post Office accounting?
Paul Rich: Yes. I don’t know if – it says, “Pathway are looking to amend”. I don’t know if they did, so I can’t comment I’m afraid.
Mr Blake: The paragraph below:
“The above has been registered with Operational Service Management, but there is a growing concern given the limited functionality and few on-line offices.”
Paul Rich: Yes.
Mr Blake: So in that period were you concerned about errors during the “limited functionality and few on-line office” period?
Paul Rich: I don’t remember it as – as I said earlier – as a large issue or a big material issue that was being brought to our attention all the time. Clearly this is an example. It’s an operational service management report which goes to me and others as part of the PDA board and when the PDA board met we would have asked what had been done about it and decided. I don’t think it’s like – it’s not sent to me to action, as it were, myself.
Mr Blake: Did somebody in particular action that?
Paul Rich: Can you show me who the –
Mr Blake: Can we look at the first page –
Paul Rich: Because I don’t remember this report. Yes, it would have been – yes it would have been – the person who would have been tasked with fixing it with ICL would have been Peter Crahan. He’s the guy in charge of the PDA at this point bearing in mind the PDA existed and Horizon programme or – it hadn’t been moved yet to Horizon or it was in the process of doing, so I think (unclear).
Mr Blake: I’m going to move on to 1998, the spring to autumn of 1998. Can you briefly tell us how your role changed in that period?
Paul Rich: In the spring/autumn ‘98?
Mr Blake: Yes. I think after 1997 did you leave the PDA board?
Paul Rich: No. I think what happened was the PDA board – the PDA started to dissolve and – that was part of the recommendations, as John said yesterday – we brought more things back in-house. There was still a residual PDA dealing with contractual matters and that sort of things, where there was a joint contract, but we brought a lot of it back into a new Horizon programme director – which was Dave Miller. So my role changed, really, to more like a commercial strategic, rather than delivery. Of course I needed to be kept appraised of where we were because clearly things are iterative to some extent.
Mr Blake: So at that stage was David Miller more focused on the technical matters and you were –
Paul Rich: On the delivery.
Mr Blake: – on the delivery.
Paul Rich: – on the delivery, including the technical matters. He would have been resourced up. We resourced him up. He would have reported directly on those matters to the managing director.
Mr Blake: And your role at that stage, you saw as more strategic?
Paul Rich: More strategic commercial I suppose if there had been major contract re-negotiations that came out but everything, then of course, was in hiatus during, as John – I hadn’t heard him use that expression before, “the year lost” – in ‘98 when the Benefits Agency and DSS basically called time and the Treasury working group was set up et cetera, et cetera.
So Dave Miller was trying to keep the programme going, with his Benefits Agency, under a guy called Vince Gaskell, who was on there, was doing that for the Benefits Agency, if you like, and we were more involved, really, in trying to cope with the various reports, consultancies and people sent to us by the Treasury working group and give what we thought was good advice.
Mr Blake: Can we look at POL00038828, please and this is moving to March 1998, March and April.
Paul Rich: Ah yes. This was the first Bird & Bird report.
Mr Blake: Do you know whose handwriting that is on the front page? It doesn’t matter if you don’t recognise it.
Paul Rich: I would have thought it’s Dave Miller’s but it says “Dave” so it’s confused me. I don’t therefore.
Mr Blake: Can we look at page 5 in paragraph 310. Again is this the same – I don’t if this is same handwriting or if this is handwriting that you recognise at all?
Paul Rich: No, you gave me a document yesterday which had annotations on it which looked similar to this.
Mr Blake: You don’t need to be a handwriting expert. If it’s not yours, then that’s sufficient.
Paul Rich: – I don’t know – Dave would have had a number, you know, people working for him and I imagine it’s one of those.
Mr Blake: So there are some concerns set out there; I’m going to very briefly summarise them. There were some concerns about paperwork things or agreements to agree, contracting authority responsibilities et cetera. The second bullet point security controls, security requirements. The third paragraph, training requirements and solutions. That seems to be crossed out, I’m not sure if you are able to assist as to why that would be crossed out?
Paul Rich: No.
Mr Blake: The fourth, problems with the Post Office estate, availability of ISDN connections.
Paul Rich: I’ve mentioned some of those before haven’t it?
Mr Blake: Yes. Then it’s the next paragraph that I’d like to look at:
“The Pathway systems interface to BA and POCL systems. Some of these are new systems, being developed in parallel with Pathway, others are being modified to include the Interfaces. There have been a number of issues with the interface systems, particularly with the BA CAPS programme and the POCL reference data system.”
Does it say – if may say “infer culpability” or something on the right-hand side, but again if it’s not your –
Paul Rich: Honestly, I couldn’t tell you who that was.
Mr Blake: Were you aware of POCL reference data issues at that stage?
Paul Rich: I know we had to get our reference data more systematic in order to be able to cope with an automated world. I wasn’t aware of, at that time – I don’t remember at any rate – reference data issues being a particular interface problem at that time. I’ve read subsequent reports that you have sent me, or the Inquiry sent me rather, sorry, that talks about reference data issues –
Mr Blake: Would you have received these Bird & Bird documents?
Paul Rich: I think I would have received this Bird & Bird document at that time. I think I referred to it in my witness statement, so I might have done. It’s confusing because this is Project Mentors and they did a separate one, out of the blue really, at the end of ‘99 as well.
Mr Blake: So who were Project Mentors, very briefly?
Paul Rich: I think they were a consultancy, an IT consultancy or claim they were. I think they were run by a professor –
Mr Blake: We may see that –
Paul Rich: – that obviously Bird & Bird, who were the joint contract solicitors for the PDA, knew.
Mr Blake: Can we look at POL00069096. That’s a meeting of the Counter Automation Steering Group on 27 March 1998. I think you’ll need to go over the page but it’s clear that was sent to you, the first page that we skipped over, and you would have received that because you’re named as being present at the meeting as well?
Paul Rich: This is the minutes of this meeting I see, yes. Thank you.
Mr Blake: Can we look at page 3 and the top two paragraphs there. It says, about halfway down the first one:
“POCL would not seek to delay Pathway’s April 1999 roll-out date, but before accepting the system would want to be certain … it was working correctly; work on EPOSS was continuing and Pathway had indicated that while it could provide a system which met the contract, its lack of robustness could generate high level of errors within POCL. This was being investigated although it was difficult to quantify how the system would work until after it had been installed and was operational.”
That all sounds quite serious at that stage, doesn’t it, especially in relation to EPOSS?
Paul Rich: Yes, I mean, that doesn’t say, though, that we would have wanted to have let this system roll out, not in my view anyhow, without the Acceptance Incidence, including on EPOSS, being cleared.
Mr Blake: So it’s similar to the evidence that you gave earlier that, although you knew there were problems it would be in the roll-out that that kind of thing might –
Paul Rich: Well, the roll-out – before going into roll out, Dave Miller I think had a system of acceptance instances which he categorised high, medium and low and there were a certain number that could be allowed but none that were high, and I would have imagined the EPOSS one was high. But that acceptance happened after my time, so I can’t really comment on that.
Mr Blake: The reference to high level of errors there sounds concerning.
Paul Rich: Well, it says “could generate high levels of errors”. It sounds like the way – this is a report by Dave, by the sound of it, and it said Pathway itself had – might have thought that the way it was doing it could generate high levels of error. So that is a cause for concern.
Mr Blake: If we look back at the first page –
Paul Rich: I mean, clearly, that wasn’t acceptable.
Mr Blake: – sorry, the second page. The attendees of that meeting: John Roberts was Chairman of that committee.
Paul Rich: He was.
Mr Blake: Would you have expected him to take that kind of information to the board level?
Paul Rich: You need to ask John that.
Mr Blake: Was your expectation at that time that those kinds of details would have been raised at board level?
Paul Rich: I would have respected John’s judgement about what he took or didn’t take to the board. This was, as he said yesterday, an extra piece of governance on the board that he included, he and his – some of his colleagues, the executive colleagues, Richard Close is the finance director, Jerry Cope is the group strategy director, Stuart Sweetman, the managing director for counters and me and Dave from within Counters – in order to more closely monitor the project.
Mr Blake: I think you have said that acceptance criteria would be an important factor in dealing with those kinds of problems.
Paul Rich: Yes.
Mr Blake: Who would you see as responsible for that?
Paul Rich: Well, the acceptance criteria, from what I’ve read recently, ICL was proposing different acceptance criteria towards the end of ‘98/’99, as I understand it, and that was then – that would have fallen to Dave Miller primarily to deal with at that point, not me, so I can’t comment on those and didn’t comment on those and those acceptance criteria, which would have then folded in, in the second half of ‘99, after the ministerial decision would have led, I imagine, to the acceptance criteria being modified, as part of the re-negotiation of the heads of terms between ICL and Post Office. I don’t know who had signatory authority within the Post Office for that, I’m afraid.
Mr Blake: You don’t know who had signatory authority but who would you have expected to have taken responsibility –
Paul Rich: For the acceptance criteria?
Mr Blake: – for the acceptance criteria?
Paul Rich: I would have imagined Dave with taking legal advice and surrounding himself with people from – who had experience of the programme and taking IT advice and others. I’m sure he would have done this, I trust Dave would have.
Mr Blake: That’s David Miller?
Paul Rich: David Miller, yes.
Mr Blake: I’m going to move on to the Treasury working group. Now, you weren’t part of that group. I think it was Jonathan Evans.
Paul Rich: Jonathan Evans, Jonathan represented Counters although I gave inputs to it.
Mr Blake: Do you think that the Post Office provided enough technical expertise to those kinds of groups?
Paul Rich: Well, there was a panel of technical experts working for the Treasury working group, alongside KPMG, who were doing the overall evaluation, and they came and talked to various people within the business. So it was really responding to them, rather than being – us saying “Here’s our technical people, could you do it?” They were asking us a lot of questions about the impact of cancellation or termination of the contract, in part or in full, and, as you heard from John yesterday, the variations to that theme got wilder and wilder and more and more radical at times.
Mr Blake: We heard earlier about the PA report which said that there was at least some lack of expertise within the Post Office when it came to those technical matters. Do you agree with that?
Paul Rich: I think we needed to improve our core competence in that. I would agree with that. You could never say you have enough capability, I don’t think. I think that would be arrogant to say so. I think we had sufficient expertise to manage the programme, given the PFI nature of it.
Mr Blake: Do you think Jonathan Evans going into that Treasury working group had enough understanding of the technical details?
Paul Rich: He would have come back and asked us about that or had provided briefings if he needed to but he was there primarily around – well, he’s a good guy but also he was the network director and a lot of the modelling being done was on the impact on the network.
Mr Blake: Do you think that those with technical expertise were given enough say on whether the Post Office should remain committed to that project at that time?
Paul Rich: Ah, I see. I don’t know. John answered yesterday, I think, that we talked about the options and Jonathan did a working group about re-tendering, if necessary, and that sounds easy, but isn’t. You know, given the situation, you had to postulate different scenarios and, indeed, I think there’s a report in the bundle where we asked – as well as asking for an external view, we asked for a peer review run by our own finance director, who was pretty fiercely independent, to say if Benefits Agency withdrew from this what was the best option. His conclusion was the least worst option was to carry on with the project.
Mr Blake: Let’s look at a letter or a note from Mena Rego. That’s POL00028649. It’s a document of 8 July 1998. Can you just tell us who was Mena Rego?
Paul Rich: Mena was one of my direct reports and she was working on – she was the Horizon development manager, I think her title was. She had a commercial background and a general management –
Mr Blake: The penultimate paragraph there –
Paul Rich: Oh, this is deep pink, yes.
Mr Blake: “… we have to get the message across very clearly to officials/Ministers that we are not a pawn in the game between DSS desire to exit and Treasury/DTI desire to prop up ICL and that if BA cease the payment Card we reserve our right to make our own decision on continuation/extension of the contract or termination and this would depend on our satisfaction on the 2 points above.”
Paul Rich: Can you remind what the two points above were?
Mr Blake: Yes, absolutely. Can we just have a quick look.
Paul Rich: Yes, I see.
Mr Blake: What did you understand by “pawn in the game”?
Paul Rich: You haven’t met Mena but she’s a very direct person and it’s probably not the language I would use. But we can’t – what she was trying to say there is that the Post Office and POCL couldn’t be the fall guy for DSS’s desire to exit and the Treasury/DTI desire to do so but put any of the losses that might result on that on to Post Office rather than ICL.
I mean, I think the background to this, which I must just mention, if I may, this was 8 July ‘98, so this is going into the Treasury working group and you will know that the KPMG report, eventually published, showed that the clear best value for money was to continue with the benefit payment card but on an extended roll-out. The only loser in that scenario was Benefits Agency and, therefore, they resisted it tooth and nail.
Mr Blake: Did the Post Office come up with a credible alternative strategy to the payment card?
Paul Rich: We talked and considered and agreed that we would migrate to a smartcard, which you will recall was in our minds at the outset and, indeed, even appears in the 1995 MOU as the system would be able to migrate to it. But it depends what you do with that smartcard and it depends on the rate of business you lose at the same time. You have to think about our post offices and our customers here.
If, as John said yesterday, it wasn’t just a technical decision it was a rounded decision, at ‘98/’99 less than 25 per cent of DSS claimants chose to have their payments made by ACT so, even when it was available to them, they were still choosing to come to post offices. We were determined that that customer choice should remain in one way or another.
So the credible alternative we tried to put up, under one of the option 2 variants, when DSS had dug their heels in with Treasury backing, to some extent, saying they wanted to terminate their part of the deal, was that we were prepared to go to a smartcard that would enable banking services in some way, shape or form, provided that BA continued its managed transition on ACT over a number of years and continue to pay Post Office and, therefore, we could pay subpostmasters and our people some amount for that card.
Mr Blake: Do you think that the Post Office weren’t thinking of a simpler plan, though?
Paul Rich: Such as?
Mr Blake: There are some criticisms certainly in some internal government reports – I don’t need to take you to them because you won’t have seen them at the time – but they were critical of the Post Office’s lack of a robust business plan at that stage?
Paul Rich: I don’t – disagree with that. I mean, I take issue with that. It’s very easy for people who don’t run things to make up comment on those things. The enablers that would have had to be in place then, as now, for that network to survive was that a universal banking product would have had to be developed. One suggestion from within the Treasury was the Post Office could become a bank to do that, which was unacceptable because of all the regulatory risks. We sort of sold our bank off ten years earlier with Girobank privatisation.
We would have had to make enabling arrangements with banks to do that and the government would have had to do that because the banks wouldn’t have been rushing to take on a lot of unbanked people who they wouldn’t have seen.
We had a plan also to provide what we called network banking services, which we then went on to develop for banks as they closed branches to come and do stuff.
That would have also required an automated network. It would have required further releases to enable things like debit cards to be done, probably an ATM-type network. So it wasn’t a quick fix, is what I would say.
Mr Blake: Can we look at POL00028644, and that’s a letter to you from David Sibbick who was the Director of Posts at the DTI at the time.
Paul Rich: Yes.
Mr Blake: That’s August 1998.
Paul Rich: Yes.
Mr Blake: At the bottom of that first page, he’s asking if Benefits Payment Card were dropped, what technology would the Post Office want and could simpler technology be used? It seems there that the Post Office are being given an off-ramp to the Horizon project if they wanted it.
Paul Rich: He’s asking for a scenario and we gave him that and I think it’s in the bundle.
Mr Blake: Did the Post Office ever come up with a simpler system as an alternative?
Paul Rich: I’d like to know what he meant by that. Such as? You know, what is – the simpler system – we had a simple system that was called payment method – paper-based payment methods, and that clearly was unacceptable and not modern. The type of technology to enable us to be competitive in the future would have had to – should build on the infrastructure that was being developed and being rolled out at that point and included the option then to upgrade it to more of a banking-type solution.
Mr Blake: I think a witness has said that, effectively, a tank was built when all you needed was a car, or something along those lines.
Paul Rich: Did they?
Mr Blake: What would your position on that be?
Paul Rich: Well, if we’d have started again without the benefits card at that time and re-tendered we would have obviously had a different type of specification, at that point, that would have taken advantage of the latest technology, without all the work that had been done on very infrastructure to start with. It wouldn’t have taken away some of the earlier ones.
It was also a quite serious point about timing, which the Treasury tried to ignore, which was – there were serious procurement law issues.
Mr Blake: I’m going to take you to a document very briefly, it’s POL00038842. It’s again about a meeting that took place with David Sibbick and it’s a question that I’m asked to ask you and I will just ask it very quickly. It relates to paragraph 2. It seems as though, from paragraph 2, there’s a proposal from the DTI for a joint spokesperson and it says “This was rejected by Paul Rich”. Do you remember that and, if so –
Paul Rich: I do remember it. I remember the background to it.
Mr Blake: Are you briefly able to explain?
Paul Rich: Hamish Sandison was the Bird & Bird lawyer who had acted for the BA and POCL in the PDA and he had – for example, we think he had commissioned that last Project Mentors report in ‘99, which remarkably and coincidentally came at the same time as DSS were considering whether they withdraw or not, and we thought he was in conflict. We had advice from – because we didn’t agree on the way forward and we had advice from Slaughters to Treasury solicitors and to our own solicitor that – I can remember the partner at Slaughters giving a very clear indication that Hamish would have been in conflict.
Mr Blake: So was your concern with Bird & Bird and the Project Mentors you just described – were you concerned that they were adopting a DSS line?
Paul Rich: The second – I’ll be honest about this and if I misremember you will have to forgive me, and you can ask Sarah Brown, but I don’t remember commissioning – jointly commissioning the second Bird & Bird report which, when I saw it, was, I think, on New Year’s Eve was sent to George McCorkell with me as a copy, which I then passed on to Dave Miller, because clearly it was technical issue, where they claimed there was insufficient requirements analysis of the Benefits Agency solution, which was a fundamental point.
But I thought it was interesting that they never raised that in their report a year before.
Mr Blake: We’ll get to that document shortly. Just in relation to this document, did you suggest an alternative spokesman?
Paul Rich: I can’t remember.
Mr Blake: Moving on to late 1998, we’re in autumn but let’s move to November, it’s POL00028421.
Mr Rich, I should ask are you okay to continue?
Paul Rich: It depends for how long because, I’m afraid, I’m of that age.
Mr Blake: I have about – I imagine I’m going to finish at half past or thereabouts?
Paul Rich: I don’t know if there will be follow-up questions though.
Mr Blake: It’s unlikely. There maybe a very short follow-up from Mr Stein.
Paul Rich: If we can say – I’ll be fine until about 20 to/quarter to.
Mr Blake: Let’s see how we do and if we need a short break, perhaps we can have a short break?
Sir Wyn Williams: Mr Rich, I want to repeat that. At any moment when you feel the need for a break, you say so and we’ll have one.
Paul Rich: That’s very kind thank you. I’d rather not lose the flow if I can help it.
That’s an unfortunate phrase, sorry!
Mr Blake: So the document in front of us is from David Miller about Horizon testing –
Paul Rich: Yes, I remember that one, yes.
Mr Blake: – and it’s the first substantive paragraph that I want to ask you about. He highlights there that:
“My present assessment is that there are some significant problems with the way Horizon passes information through to TIP. These relate to the provision of balanced outlet cash accounts and the processing of the ensuing information via TIP. Whilst we allocated some extra time during the Corbett review to sort out any outstanding issues we need to be aware of a potential threat to 14 December date.”
Paul Rich: Yes, I’m glad he wrote that and I’m glad he wrote that there will be a potential threat to the 14 December date, rather than saying we’re going ahead with them.
Mr Blake: I’m going to move on because there’s a theme developing in November 1998. Let’s look at POL00028320. This is the “Transformation Steering Group Progress Report to 23 November 1998” and can we look at page 6. There’s what is called “Red Light Issues”, which –
Paul Rich: This is another document that I only saw at 2.00 yesterday.
Mr Blake: Okay. If you need more time to consider it –
Paul Rich: This is the most substantive one, or one of the most substantive ones, so my – this is the annotations that I said I didn’t recognise. So I don’t know who’s writing this. It also said, I think at the beginning of it, that the meeting didn’t take place.
Mr Blake: So is this a document that you think you would have received at the time?
Paul Rich: I can’t tell. I would have received it if it was going ahead because I would have been chairing the meeting.
Mr Blake: If we look at the first paragraph, and that’s the only paragraph I want to take you to, “Red Light Issues, Horizon System”:
“There are major concerns about the test results emanating from Model Office and End to End.
“The results indicate that cash accounts and transaction data delivered to POCL’s downstream systems lack accounting integrity, all of which raises serious doubt about Pathway’s ability to enter into the next phase of Model Office and End to End testing without some form of remedial action.”
Paul Rich: That’s basically reflecting what Dave Miller said before, isn’t it?
Mr Blake: I was going to say, even if you didn’t necessarily see this at the time, were those issues that you would have been aware of?
Paul Rich: Yes, because Dave told us. What does whoever wrote it, said about it, may I just …
Mr Blake: If we could scroll down and highlight that?
Paul Rich: “Remedial analysis has taken place [following] meeting TIP and Reference Data personnel. Remedial action is now underway. The point must be made that we will not enter the final phase of testing until we are content that we have a robust set of code.”
Mr Blake: So somebody’s written that at the bottom?
Paul Rich: I think that’s Dave Miller’s number 2.
Mr Blake: Who was that?
Paul Rich: Could have been one of two people. My guess is – there’s a lady called Janet Topham at the time.
Mr Blake: That paragraph on the Horizon system, though, that background 23 November 1998, I’m going to now look at POL00038829 and I think this is the controversial document, perhaps, that you were talking about from Bird & Bird?
Paul Rich: Yes.
Mr Blake: Can you assist us with that handwritten note or –
Paul Rich: That handwritten note is from Mena’s secretary.
Mr Blake: “Dave” being?
Paul Rich: Miller.
Mr Blake: Thank you. Can we look at the report itself, paragraph 1 –
Sorry, there are a few different versions of this document.
Mr Blake: Sir, perhaps we could take a five-minute break now for everybody’s convenience and we can go back on in five minutes. Thank you very much.
Paul Rich: Thank you.
(12.17 pm)
(A short break)
(12.22 pm)
Mr Blake: Thank you very much, sir, we are back and I have found the relevant page, page 3 of that document. POL00038829. This is a letter from Bird & Bird, December 1998, can we look at that first paragraph, please. It’s difficult to read:
“As you will see, [Andrew Davies’] team have documented a further specific failure by ICL Pathway to follow good industry practice in meeting the Authorities’ requirements.”
Can we go to page 5. There’s a letter there to yourself and George McCorkell and Pat Kelsey from Bird & Bird, and that first paragraph summarises the view of Andrew Davies of Project Mentors. Can we just have a look at that first paragraph, sorry. The quote there is quoted from Andrew’s letter:
“… ‘deeply concerned that their findings show a serious problem with the way in which ICL Pathway have developed the system. The impact of this is likely to be that there will be failures to meet essential user requirements, causing the need for extensive rework before the system can be accepted and, potentially, operational problems if the system is rolled out.”
That’s quite a concerning statement, isn’t it?
Paul Rich: It’s his opinion, yes.
Mr Blake: The impression that it gives is that there are real concerns about Horizon at the time.
Paul Rich: But if you look at – no, not but. I think this is commissioned on the way, if I’m correct, the benefit encashment service was developed and the focus was on that, and the claim then by Mr Davies was that the same must apply to all other components. I think it also – I’m not a – you know, I’m not a technical person, I’m not an IT expert and wouldn’t ever aspire to be but I think it completely ignores the fact it was procured under private finance.
Mr Blake: Can we go to page 6, which is the letter from Project Mentors to Hamish Sandison and over the page to page 7, and it’s the second paragraph there. He says:
“Our experience of systems where requirements have not been analysed satisfactorily is that the system fails to meet the users’ needs. An effective acceptance test will identify many such failings necessitating considerable rework. The result is a significant extension of time and cost required to complete the system and roll-out it out. The alternative is to allow unacceptable processing in the operational environment, with unpredictable and potentially damaging results.”
Paul Rich: Is there a question?
Mr Blake: I’m going to move on to show you the substantive report. That’s at page 8. Can we go to page 11. Can we look at paragraph 1.3, “Scope”, the second paragraph:
“We have to date only considered the [Benefits Payment System]. Further work has recently started to perform a similar assessment of the approach adopted for other elements of the system, such as EPOSS. Nevertheless our findings are, in our view, sufficiently serious to bring into question the whole of Pathway’s design process.”
Moving on to page 14, paragraph 2.3.4. Again, at the bottom:
“Of particular concern is the EPOSS system [that’s the second paragraph]. We are informed that at a relatively early stage Pathway wanted the Authorities, principally POCL, to be involved with the design of this element. The plan was to use the Rapid Application Development … methodology to design the system. This approach was started, but discontinued after some months, when the Pathway staff member involved left the project. The suggestion to use RAD leads us to believe that more traditional methods have not been used, and since the RAD experiment was abandoned, we have doubts whether any proper requirements analysis has been performed.”
On the same page. Can we go to page 14, it’s 2.4.3. Sorry, that’s the part we just read. Then over the page again. You have addressed this in your witness statement and you say that you were only copied in to this document and that you would have passed it to others.
Paul Rich: Yes.
Mr Blake: You say:
“My recollection is that POCL did not necessarily agree with the report as proving the system was fundamentally, technically flawed.”
So, again, that’s the use of the term “fundamentally flawed”. Were these issues raised here serious issues?
Paul Rich: Clearly, they were serious – seriously phrased. I passed this on to Dave Miller at the time because he was clearly in the process. As I said, this is December ‘98 when he was considering whether or not to authorise the further release. As this says, it doesn’t actually analyse the EPOSS system. It talks about things had started to go on the EPOSS system, it talks about rapid application development in the past, which ICL did want to use – RAD, it’s called – and we didn’t particularly want them to do that unless it had quality of outputs to do so.
As I said before and I stick to that, the previous Project Mentors report a year or so earlier on the whole thing seemed not to address the issue of insufficient requirements analysis at all, which I find surprising, and I just note that it came out as the DSS were considering praying in aid whether or not to continue with the benefit card or not.
Mr Blake: Coming in a year later, presumably that’s even more concerning because you are further down the line and these issues are being experienced?
Paul Rich: But – yes, but if it was a fundamental design flaw because of insufficient requirements analysis by ICL, that would have been apparent earlier on –
Mr Blake: Now that –
Paul Rich: – logically.
Mr Blake: Now that they have found these issues, whose responsibility within the Post Office would it have been to take those forward?
Paul Rich: To consider it?
Mr Blake: Yes.
Paul Rich: Dave Miller.
Mr Blake: You said earlier that, at this stage, I think you were dealing with strategic matters and David Miller was dealing with the more technical matters?
Paul Rich: He was dealing with delivery. I was in my last – as I said earlier on, I was appointed to a role completely outside of the Post Office Network when the entire group reorganised itself, as a managing director of a new unit in March.
Mr Blake: Who would you have expected David Miller to have brought this to their attention?
Paul Rich: Maybe Stuart Sweetman. I don’t know.
Mr Blake: Do you think it was sufficiently serious to bring to the managing director’s attention?
Paul Rich: You need to ask Dave that.
Mr Blake: If you had received it at the time, would you have brought it to the managing director’s attention?
Paul Rich: Depends in what context. As I said, I would have probably talked at length to the person authoring the report first to understand it because, as I said, I had reservations on it and I seriously don’t remember commissioning, even though, no doubt, we’d have paid half of it.
Mr Blake: Do you think that by the end of 1998 and the beginning of 1999 technical issues were being taken seriously enough within the Post Office?
Paul Rich: They would have been taken seriously. I maintain, as I said, we would not have been expedient for the sake of speed or trying to shoehorn a solution that suited us strategically if we didn’t think it could work. We relied on – you’d have seen Dave Miller’s notes to us. So we knew it, “no denial”, as I said. I know I’m sounding like I’m in denial about this report. It’s the first time probably today but I’m irritated by it.
We then – you will, no doubt – well, maybe you won’t, there’s another document that Dave Miller wrote in April to Vince Gaskell. That’s POL00028407, where he says the team – following these issues, the team now believe they’re ready to be able to roll out, and I would have relied on Dave’s judgement about that, standing from afar. As I said, I was in the process – I had a new job but I was hanging on for a couple of months because the ministerial decision didn’t come until May, so I was still giving input at that point.
Then the next I saw was something in a note from Keith Hardie, where I was a copy amongst many others, as I said in my witness statement, that the Post Office was beginning to roll-out Horizon. That’s POL00028463.
Mr Blake: Let’s stay in January 1999 for now and just look at POL00031230. This was a report by Post Office’s POCL’s finance director.
Paul Rich: This is the one I mentioned earlier.
Mr Blake: Yes. What was your involvement in this particular document?
Paul Rich: Well, there was an earlier – we called these peer reviews. There was an earlier one done in ‘97 by the strategy director of Post Office. This was done by the finance director – sorry, Post Office Counters, I should say – and we asked him, as a contingency, to look at, as I said earlier, whether or not – without me getting – I mean, he’d have talked to me and given input but without me trying to influence his decision or his analysis, in any way, shape or form, really, to give an opinion on what the best way forward for POCL would have been if this scenario occurred.
Mr Blake: Can we look at page 2, paragraph 2.5 and 2.6:
“Even on the basis of protecting benefit payments, the go/no go decision is finely balanced, with neither option being fully satisfactory for POCL. Proceeding means full commitment to an automation route and a partner, neither of which are ideal in the context of realising the new Counters vision. But not proceeding would so delay the building of automated capability, and undermine the business’ credibility internally and externally, as to put the vision at significant risk of becoming undeliverable.”
Paul Rich: I think I said earlier it was the least worst option.
Mr Blake: “Several senior managers, close to the project, but no principal negotiators, whose judgement I respect, express significant reservations about the risks of proceeding. These centre of their continuing doubt about the ability of ICL to deliver a satisfactory product; the absence of transparency in the PFI contract; the risk that ICL’s financial fragility will endure throughout the project, with the possibility of repeated claims on The Post Office for extra contributions (which, by then having no alternative, it will be unable to resist); and doubts about POCL’s own ability to give it the focus essential for success.”
Paul Rich: Yes, none of that is new news, though, is it, really? I mean, the track record demonstrates that.
Mr Blake: Over the page, the decision is:
“On balance, I agree that it remains right to press ahead with Horizon, despite the extra costs involved.”
Did the commercial importance of seeing Horizon through at that stage outweigh the kinds of technical concerns that we’ve talked about this morning?
Paul Rich: No. I mean, as I said, the decision was in a round and a business decision always is surrounded by both technical, commercial, operational and financial aspects. That’s a matter of judgement for general managers but we would not – I don’t believe – well, I find it really hard to believe that anyone from that culture, at that time, would have compromised quality knowingly, in order to be expedient to suit strategic or financial matters.
Mr Blake: I’m only going to ask about a couple more documents. The first is a return to your witness statement, WITN04030100 and it’s page 33, paragraph 94.
Paul Rich: Yes. 94? Okay.
Mr Blake: Yes, I’m just going to – can we bring that on the screen? Thanks.
So what you have said there is you were less involved in the Horizon project as 1999 proceeded and, by May 1999, you were no longer involved?
Paul Rich: At that time, yes.
Mr Blake: Can we look at POL00021469.
Paul Rich: You are going to show me a document where I was, okay.
Mr Blake: Well, it’s a board meeting.
Paul Rich: Is this the one in 2000?
Mr Blake: Yes.
Paul Rich: That’s me going, though, in my new role as MD customer management and the board asking – I’m an attendee, aren’t I, along with other people like, I don’t know, someone else from – yeah, Basil Larkins, the managing director of Network Banking. You see, we all had different titles then because we are one of the 17 managing directors under it.
Basil was the Network Banking person. I was the managing director of customer management and, within that customer management, there was a peripheral – well, not peripheral, but a smaller part of the job was to look at the opportunities for electronic government with the Post Office in general, including perhaps at counters.
Mr Blake: You would have been present for the whole meeting there?
Paul Rich: I doubt it.
Mr Blake: Would you have stayed for the part that addresses Horizon?
Paul Rich: I doubt it. I mean, the normal way – I don’t know. I can’t remember but the normal way of board meetings, if you weren’t a board member, was that you attended for the item in question that you were asked to present.
Later on, when I was doing the six-month job as the acting MD and I attended board meetings, I would have been present at those board meetings. But that’s later than that.
Mr Blake: Can we look at page 7, the bottom of page 7. I think this is a document that we went through yesterday and I’m not going to spend any time on it but this is the part where discussions turned to the commercial development of Horizon and things had moved on. Were you aware, at least at that time, that things had moved on to commercial exploitation?
Paul Rich: Yes. I mean, I’d have been away from Post Office Counters Limited entirely, or the three business units that it had been split into by then, including Post Office Network, which Dave Miller was the managing director of, and he would have – you know, we’d have met from time to time and he’d have said, “Well, we’re now rolled out to 4,000 offices”. But clearly at that point, I certainly was completely unaware of any material accounting or balancing-type issues at that point and, therefore, my brief was, as part of my new job, to have a team looking at Government Gateway opportunities.
Mr Blake: Who, if anybody, would you have passed on your knowledge to about those technical issues that you gathered –
Paul Rich: Oh, Dave Miller, obviously. We had that large overlap, didn’t we, and there would have been quite a lot of continuity in the team from the team that were brought back from the PDA in to work for Dave. I mean, clearly Stuart Sweetman. You’ll be talking to Stuart, I think, later on. Stuart was still the group MD for those matters that included Post Office Network and Network Banking so he will have had a rounded view about Horizon.
Mr Blake: If there were concerns about Horizon at that stage – so March 2000 – who do you think would have or should have raised them at the board level?
Paul Rich: Well, I imagine the way it works, certainly when I was managing director of customer management, Stuart was also my group MD then, who sat on the executive board alongside John, you know, as the chief executive, and we would have had regular contacts on how the big issues in my patch were going and if Stuart thought they had been serious enough or if I’d have proposed that I wanted board support, I would have expected the channel to go through him.
Mr Blake: Finally from me, I think you have asked to very briefly address the Chair in respect of your overall reflections.
Paul Rich: Yes, Chair. All I wanted to say was, really in line with my witness statement, that I fully respect this Inquiry and really hope that you get to the bottom of it. I have been reflecting hard ever since I’ve been asked to come here as a witness, back in May, with all the documents and I want to express my own sadness about the impact this has put on so many lives.
In some of the stuff I’ve seen, either through the press or through recent documents that I’ve seen from later periods, I’m frankly baffled and shocked by how some of these people were treated and I’m completely baffled and it’s not a culture I personally recognise in terms of the ethos of what must have been happening at that time or the governance that allowed it.
So that’s all I wanted to say.
Mr Blake: Thank you very much, Mr Rich.
I’ve just been given one very brief question that touches on earlier matters and it relates to PFI. You have, throughout your evidence, referred to the difference that PFI made to the process. Can you very briefly just tell us what you mean by that.
Paul Rich: Well, contractually, it meant, as I said, the risk was – for design, operate, build and run was with the supplier, not with the procurer. In addition, because of this particular PFI, that also included the risk around benefit fraud occurring at post offices, which I’m not aware of any other PFI-type project in the world that did that.
It also meant that the supplier only got paid on outcomes and outputs. So until – for example, as I understand it, until the system was accepted before roll-out, there were no substantial real payments to ICL so, clearly, their cash flow projections from the start of it would have been completely up the wall, as we heard earlier from their parent company to John yesterday.
The only other thing – two other things I’d say. One is that, therefore, I think it was a learning experience because the nature of this PFI project was unusual. It was usually used by Government for capital projects that were properties or leases, or something like that, where there’s some more secure – it wasn’t really operational. So we were learning about the boundaries between being able to go in and assure ourselves of the details of the design, where ICL would have said, “No, under PFI you don’t do that, wait for the outputs and test them”.
The final thing I’d say is that, of course, this project straddled two governments. So PFI was a Tory policy, you know Treasury guidance in the late ’80s. New Labour came in, middle of ‘97, and so there was some concern, certainly among subpostmasters and others and no doubt ICL, about what the attitude to a PFI project would be, whether it would be within policy for them to do it and I think it morphed into what New Labour called a public sector/private sector partnership.
So I think it got re-badged, really, and they early on committed – I can remember reading the document in the bundle with a meeting with subpostmasters, with the Federation actually, with a minister assuring them that it wouldn’t be pulled on the grounds that it was a PFI project.
Mr Blake: Sir, I believe there may be a question or a short series of questions from Mr Stein; is that correct?
Mr Stein: Sir, yes.
Sir Wyn Williams: All right. Yes, ask the questions please.
Questioned by Mr Stein
Mr Stein: Thank you, sir.
Mr Rich, my name is Sam Stein, I represent a number of postmasters, mistresses and managers in relation to what happened through the use of the Horizon System.
Paul Rich: Yes, Mr Stein.
Mr Stein: You have just provided some answers to Mr Blake that touch upon the question of PFI?
Paul Rich: Yes.
Mr Stein: When you were giving your evidence earlier today, you spoke about the Bird & Bird document you were being referred to and you made this comment that it completely ignores the fact that it was under a PFI. Does that mean that the burden of governance running and general maintenance of the Horizon System was placed upon Fujitsu?
Paul Rich: Primarily, yes, but I was also saying it in relation to the supposed innovation that they could bring in designing and developing the system. So a way of – the report talked about a requirements analysis that was allegedly standard industry practice, which may or may not have been true, I don’t know. I don’t know if that was an outdated one but my guess is that ICL would argue – I’ll let them argue for themselves – that they did do sufficient requirements analysis once they did it and they applied an innovative way of developing.
Mr Stein: You also make some general comments that you set out, regarding your bafflement regarding what happened to subpostmasters. I’m just going to remind you of a part of your own statement at paragraph 116, page 39 of your statement. You say this:
“I am truly baffled by the apparent later professional advice, investigative processes and governance that appears have led to so many unjust prosecutions.”
Can we unpick that please. What the later professional advice that you’re referring to?
Paul Rich: I’m only reading what I’ve read in the media and in the judgment that was – that was in the opening counsel’s statement, that it would seem – I don’t know. I mean, Post Office Limited will now speak for itself but it would seem that it was ill-advised in going ahead. The investigative processes seemed to be ones that I didn’t recognise from my days. Certainly, non-disclosure would have been a real issue to me with any investigative processes around fraud. The governance processes, I’ll leave that to Post Office Limited and perhaps UKGI because I can’t see how a board would have knowingly – knowingly – not noticed that 700 people had been prosecuted cumulatively, roughly.
Mr Stein: Regarding you mention in your statement and just now of the investigative processes, and also mention of governance, whilst you were at the Post Office and dealing with matters up until, I think, 2002, where you moved over to the RM group more generally until 2005, what control was put in place in relation to the investigative processes as regards subpostmasters, mistresses and managers?
Paul Rich: Well, the investigative – by the way, I wasn’t involved continually until 2002, just to be clear on the records.
The investigative processes were largely as John described yesterday, really. You know, the investigative processes were done separately and at arm’s length but if there’d have been a high – a sudden hiatus or peak of them I’m sure it would have been brought to the attention through the relevant senior manager who would have been in control of that.
Mr Stein: You say that they were done at arm’s length. What does that mean to the disclosure process of bugs and errors within the Horizon System? Were those bugs and errors brought to the attention of the arm’s length body dealing with the investigation of subpostmasters and, if so, how?
Paul Rich: I don’t know because, at the time I was there, I don’t think there were bugs and errors – I don’t know. I absolutely don’t – I can’t be absolutely certain but, by the time I left in this period, in this phase, in March ‘99 I don’t think – you might correct me – I don’t think in the live trial offices there were any prosecutions brought.
Mr Stein: Well, I could correct you. There were bugs and errors in place –
Paul Rich: No, I didn’t say that. I didn’t say that. I didn’t say there weren’t bugs and errors. I said bugs and errors that led to investigation and prosecution.
Mr Stein: Mr Rich, my question was in relation to what controls were put in place to make sure that bugs and errors were brought to the attention of the investigation processes. What can you help regarding that?
Paul Rich: Well, we wouldn’t have – during that time if there were bugs and investigations that we thought had technical issues related to them, which was the point of the live trial, as I said, and therefore needed fixing, we wouldn’t have put those to the investigation department.
Mr Stein: How did the investigation department get to learn about bugs, problems, issues with the Horizon System?
Paul Rich: I don’t know.
Mr Stein: What’s the system?
Paul Rich: I don’t know.
Mr Stein: Who’s in charge of it?
Paul Rich: I don’t know.
Mr Stein: Who should have been in charge of it?
Paul Rich: I don’t know. I can’t help you.
Mr Stein: Thank you. Thank you, sir.
Sir Wyn Williams: That concludes the questioning, I take it, and, assuming that’s the case, thank you very much, Mr Rich, for making a detailed written statement and also for coming to give oral evidence. I’m grateful to you.
Paul Rich: Thank you. Thank you for the opportunity.
Mr Blake: Thank you, sir. It’s now lunchtime. Could we come back at – would it cause anybody inconvenience, including yourself, if we came back slightly earlier, perhaps at 1.50?
Sir Wyn Williams: No, I was going to suggest that we (a) make a determined attempt to finish Mr Copping this afternoon and, therefore, (b) if it helps to have a shorter lunchtime we should do so.
Mr Blake: Excellent. Thank you very much, sir. We’ll come back at 1.50.
Sir Wyn Williams: Fine. See you all then.
(12.55 pm)
(Luncheon Adjournment)
(1.50 pm)
Ms Hodge: Good afternoon, sir, can you see and hear me? We can’t hear you. You appear to be on mute.
Sir Wyn Williams: Well, I didn’t think I was on mute. Am I on mute now?
Ms Hodge: No we can hear you perfectly, thank you.
Sir Wyn Williams: Fine. Our next witness is Mr Copping.
Peter Copping
PETER JAMES COPPING (sworn).
Questioned by Ms Hodge
Ms Hodge: Please give your full name.
Peter Copping: Peter James Copping.
Ms Hodge: You should have in front of you a witness statement, dated 2 September of this year.
Peter Copping: Yes.
Ms Hodge: Could I ask you please to turn to page 18 of your statement. Do you see your signature there at the end of the statement?
Peter Copping: Yes.
Ms Hodge: Is the content of the statement true to the best of your knowledge and belief?
Peter Copping: Yes.
Ms Hodge: Mr Copping, your statement and its exhibits are now in evidence before the Inquiry. I would like to begin by asking you a few questions about your professional background. You qualified as a chartered engineer and a Fellow of the Institute of Engineering and Technology; is that correct?
Peter Copping: That’s correct.
Ms Hodge: What competencies were you required to demonstrate to qualify as a chartered engineer?
Peter Copping: It’s a long process but, in essence, you have to display technical competencies, managerial competencies, in quite a wide range of topics.
Ms Hodge: You’ve explained that you worked in the electronics and telecommunications industry for approximately ten years before joining PA Consulting; is that right?
Peter Copping: That’s correct.
Ms Hodge: PA Consulting being a management information and technology consultancy?
Peter Copping: That’s correct.
Ms Hodge: You joined that organisation in 1976?
Peter Copping: Yes.
Ms Hodge: Before being appointed a director of PA Consulting in 1990, you worked on a variety of telecommunications and information technology projects; is that right?
Peter Copping: Yes.
Ms Hodge: Did these projects require you, at any stage, to undertake software design and development?
Peter Copping: Personally, no, but I did lead teams that were doing that.
Ms Hodge: How would you characterise your area of expertise in engineering?
Peter Copping: Broadly speaking, I would characterise it as in the telecommunications area, networking and IT.
Ms Hodge: You first became involved in Horizon when you were commissioned in the summer of 1997 to leader of the PA Consulting of what was known at the time as the Benefits Agency and Post Office Counters programme; is that correct?
Peter Copping: That’s correct.
Ms Hodge: Had you ever previously worked on a project of the scale and complexity of Horizon?
Peter Copping: Not quite the same. I certainly worked on large projects of similar scale in the mobile telecommunications area in particular.
Ms Hodge: Did you have any prior experience of working on an IT system developed by ICL?
Peter Copping: No.
Ms Hodge: I’d like to, if I can, briefly explore what you understood at the time about the broader context of the review that you were asked to undertake. Why had that review been commissioned?
Peter Copping: Sorry, could you repeat that?
Ms Hodge: Why had your review, in the summer of ‘97, to your understanding, been commissioned?
Peter Copping: Primarily because of delays to the project.
Ms Hodge: What had arisen as a result of those delays?
Peter Copping: There were concerns about the possibility of future delays, there were concerns about Pathway’s ability to deliver and there were concerns about Post Office readiness to accept Horizon.
Ms Hodge: In your statement you describe the purpose of the review as being to identify the reasons for the delay to the project and to recommend actions to de-risk the project to bring it back on track; is that correct?
Peter Copping: That’s correct.
Ms Hodge: You were also required, were you not, to make an assessment of the programme’s future delivery capability?
Peter Copping: That’s correct.
Ms Hodge: That assessment involved examining not only management and resourcing issues but also the technical aspects of the project, which had a bearing on the programme’s ability to deliver its end-to-end delivery obligations?
Peter Copping: That’s correct.
Ms Hodge: You’ve explained in your statement your review focused on four principal areas. These were the business objectives of each stakeholder; the contractual arrangements between the parties; thirdly, the programme management processes; and, finally, the technical infrastructure proposed for Horizon by ICL Pathway. Is that right?
Peter Copping: Yes.
Ms Hodge: You use a term “technical infrastructure” in your statement. Can you explain what you mean by that?
Peter Copping: Essentially, that is the hardware platform on which the software services reside from the counter back into various back-end systems.
Ms Hodge: Are you suggesting, therefore, that you were only asked to consider the hardware, as opposed to the software?
Peter Copping: No, no.
Ms Hodge: Elsewhere in the documents we see the term “technical architecture” used. So far as you’re concerned, are they one and the same: architecture and infrastructure?
Peter Copping: The architecture defines the various layers in the system that worked together to make up the infrastructure.
Ms Hodge: So slightly different nuances then?
Peter Copping: Yes.
Ms Hodge: Do you recall how far the project had progressed at the point at which you undertook your review in the summer of 1997?
Peter Copping: When we started work there was a – I think we were presented with a programme rework which was titled “Version 3”, and I think all of our work was based on that particular document.
Ms Hodge: It might assist if we bring that up. That’s POL00028186, please. Is this the document to which you were referring?
Peter Copping: Yes.
Ms Hodge: The “Programme Delivery Authority Master Plan Version 3” –
Peter Copping: Yes.
Ms Hodge: – dated 8 April 1997. Could we turn to page 8, please. You see there a number of strategic milestones in the project. Can you see that in front of you?
Peter Copping: Yes.
Ms Hodge: The first of which was the initial Go Live implemented in one post office on 23 September ‘96 and then rolled out to ten post offices on 23 October ‘96?
Peter Copping: Yes.
Ms Hodge: That was followed by the roll-out of Pathway infrastructure on 7 March ‘97, so we see next to B1?
Peter Copping: Yes.
Ms Hodge: And B2, the release of what became known as software release 1b, which, as we can see, implemented OBCS functionality. Can you describe what OBCS was?
Peter Copping: It was order book CS, something. I don’t remember.
Ms Hodge: The control service?
Peter Copping: Control service, that’s right, yes.
Ms Hodge: Were you aware of what its function was?
Peter Copping: This was the service that was used to confirm that the person in the Post Office was entitled to the benefit that was on the order book, as I understood it at the time.
Ms Hodge: We can see a further date of 30 June ‘97 about midway down the page and that was the planned release of Pathway Release 1c, which was due to contain further OBCS – so the order book control service – and BPS, which was the Benefit Payment Service, functionality. We know, however, that milestone had been missed because, at the point at which you conducted your review, development work on Release 1c was ongoing. Is that consistent with your recollection?
Peter Copping: That is correct.
Ms Hodge: What did you understand at the time about the state of development of the Post Office counters functionality, by which I mean EPOSS, the Electronic Point of Sales Service, and APS, the Automated Payment Service?
Peter Copping: My goodness, I really don’t remember.
Ms Hodge: In your statement you explained you adopted two principal methods of assessment, the first conducting a series of in-depth interviews and follow up investigative meetings with senior figures in each of the stakeholders, those being the Programme Delivery Authority, Post Office Counters, the Benefits Agency, Pathway and ICL; is that right?
Peter Copping: That’s correct.
Ms Hodge: The other aspect of your review or your assessment was a document review essentially; is that right?
Peter Copping: Yes.
Ms Hodge: In that you reviewed a significant amount of documentation relating to technology status and plans for Horizon?
Peter Copping: Yes.
Ms Hodge: If we could start by addressing the second of these methods, your document review, could you please describe the types of technical documentation to which you were granted access to ICL Pathway?
Peter Copping: First of all, we would have started with a demonstration of the model office system. We would have taken presentations from ICL on the software status, status of development, the overall architecture, the way the system was supposed to work and their view of the current issues in the program.
Ms Hodge: What you’ve just described there, it sounds mostly like a practical demonstration and oral presentations, rather than an analysis or a review of documents. Did you carry out such an analysis?
Peter Copping: Yes, there were analyses undertaken by members of the team on technical documentation, mostly in the software area, also with Escher.
Ms Hodge: So it was other members of your team, employees of PA Consulting, who looked at the more technical aspects?
Peter Copping: Yes.
Ms Hodge: Do you recall what, if anything, they told you about the completeness or quality of the design documentation that was shown to them?
Peter Copping: Sorry, I don’t recall that level of detail, I’m afraid.
Ms Hodge: If we turn back to your first method of assessment to which you referred in your statement, the interviews which you conducted, you described carrying out more than 30 face-to-face meetings and interviews; is that right?
Peter Copping: Yes.
Ms Hodge: Do you recall the names of those whom you interviewed?
Peter Copping: They are all listed in the report.
Ms Hodge: Could we possibly bring that back up POL00028092. I say “bring back up”, this is for the first time. Thank you.
At the conclusion of your review, you prepared a written report; is that right?
Peter Copping: Yes.
Ms Hodge: The third and final version of that is dated 1 October 1997.
Peter Copping: That’s correct.
Ms Hodge: That’s the report to which you just referred?
Peter Copping: Yes.
Ms Hodge: If we could turn to page 48, please, of the report, we can see here Appendix A, a list of those whom either you or your colleagues interviewed in connection with this review; is that right?
Peter Copping: I probably met most of the people on that list myself at some stage, either in individual meetings or group meetings with ICL Pathway particularly.
Ms Hodge: You’ve explained in your statement that some of the in-depth technical interviews were attended by specialists employed by PA Consulting; is that right?
Peter Copping: Yes.
Ms Hodge: Why did you consider it necessary to bring in specialists to conduct those technical interviews?
Peter Copping: It’s a way of working to ensure that we cover the ground appropriately.
Ms Hodge: Did you yourself have the necessary expertise to deal with the more technical aspects of the project?
Peter Copping: Did I?
Ms Hodge: Did you have the necessary expertise to deal with the more technical aspects of the project?
Peter Copping: On the networking and architectural issues, yes, but on the software aspects and, particularly in regard of Escher and the processes ICL were using for development, no.
Ms Hodge: You were therefore reliant upon your colleagues?
Peter Copping: Correct.
Ms Hodge: In your report, you identify a number of concerns about technical issues with Horizon, which were raised by senior figures in the Benefits Agency, in Post Office Counters and Pathway. Do you recall the nature of those concerns?
Peter Copping: In the report?
Ms Hodge: Yes.
Peter Copping: I’d have to read the report again.
Ms Hodge: If it assists, at page 28, please. Thank you.
At the bottom of page 28 there’s a paragraph 3.3.5 entitled “Technical issues”. It records:
“Concerns have been expressed to us about the ability of the solution to meet the security requirements, whether it is scalable to support a 40,000 terminal network and what performance will result. Concern has been increasing with failures in test and by regular requests by Pathway for exclusions to key releases, mainly concerned with security features.”
Is that consistent with your recollection?
Peter Copping: Yes. The security issue was a particularly difficult one, I think, for Pathway because I would say there were so many moving parts. My recollection is that the security requirement was made increasingly more demanding as it became aware of the risks and the risk transfer arrangements in the PFI contract to ICL.
Ms Hodge: Could we please turn back to page 8 of the report, where we see a part of your management summary. In relation to Pathway, at M3.4 – excuse me, if we can scroll down a little bit – you’ve, observed here, in the bottom paragraph:
“We believe the current status of the Programme is surrounded by considerable contractual ambiguity, Pathway are, in essence, proceeding on an ‘own risk’ basis to deliver Release 1c with a ‘known problems register’ and its proposal is to address the ‘known problems’ in Release 2.”
Were you shown a copy of the known problems register?
Peter Copping: I’m sure we were.
Ms Hodge: What did you understand the purpose of the register to be?
Peter Copping: This would have been issues that were expected to take longer to resolve than the plan allowed and, therefore, they’d be deferred into a subsequent release.
Ms Hodge: When you say “issues”, what types of issues did you understand?
Peter Copping: Development releases, essentially, that would take longer to work through.
Ms Hodge: Are we talking about problems in the software, bugs and errors and defects, things of that nature?
Peter Copping: Not necessarily problems. More likely an underestimation of the effort and time required against what I recall was, in some instances a moving requirement over time, and I would put the security requirement into that category, for example.
Ms Hodge: So your understanding, essentially, was these were generic problems with the software release, rather than specific issues that had been identified?
Peter Copping: I think that’s a fair description, yes.
Ms Hodge: In your report, you identified another significant concern on the part not only of the sponsors but also of Pathway, which related to the robustness of the technical architecture; is that fair?
Peter Copping: Yes.
Ms Hodge: Can you describe the nature of the concerns which were articulated to you?
Peter Copping: At the time we did our review, my recollection is that Pathway’s approach to testing was to test individual components and then fit them together and retest and it was at that stage that I think we detected there were certain concerns that, when everything was put together, it might not be as robust as perhaps was expected.
Ms Hodge: Do you recall whether the concerns expressed to you related to any particular component or whether it was a more general concern about the overall architecture?
Peter Copping: There was a particular concern about Escher and, if I recall correctly, Pathway did institute a rework of Escher’s software, as a result of the issues they were experiencing there.
Ms Hodge: What steps did you take to investigate these particular concerns that had been articulated?
Peter Copping: We would have pursued them with further investigations of any documentation that was available and further face-to-face interviews.
Ms Hodge: What conclusions did you ultimately reach at this stage about the robustness of the architecture?
Peter Copping: That it would be – I think we took the view that the overall system was achievable in development terms. The question was how long it was going to take before it became completely reliable and robust.
Ms Hodge: I’d like to turn now to your findings about the causes of the chronic delays to the programme. In your statement you describe the contractual arrangements between the parties as being a significant cause of initial delays to the programme. Can you explain the basis of that conclusion?
Peter Copping: The contract was let under a private finance initiative which is where the risk of delivery is transferred to the supplier and this particular contract was unusual, in that there were differing business objectives between the sponsors, and that created a lot of tension between the parties – and I think I’ve lost the thread of where I was going on that.
Ms Hodge: That’s okay. So I think you’ve explained that, essentially, much of the detailed specification for the contract hadn’t been agreed at the point at which –
Peter Copping: Well, there were a lot of agreements to agree and I never was really sure why that was allowed to happen and, normally, one would expect, in a PFI contract, that the supplier would be allowed to work up, once the contract has been let, the proposition to deliver for the outline requirement.
But because of the agreements to agree arrangement, it took a long time for ICL to work through each individual part of the system to decide what actually needed to be delivered and, at one point, I recall they proposed to use a rapid application development method, which was becoming – just coming into favour around that time, which allowed the developer to work closely with the sponsoring party to explore how an application might work and I think ICL were unsuccessful in pursuing that particular approach because of the reluctance of sponsors to become engaged.
Ms Hodge: Did you consider the rapid application development technique to be suitable for a project of this scale and complexity?
Peter Copping: I couldn’t see a reason why it wouldn’t be, provided the parties were happy to pursue that particular approach. It’s probably worth saying that there were many occasions where we were coming to a view that this was being treated by the sponsors and particularly the Benefits Agency as a supply and build contract rather than a PFI contract where there was a lot of intervention from the sponsors because they weren’t necessarily happy, for reasons of their own, with what was going on in the development activity and that in itself caused delays.
Ms Hodge: Another factor which you identified as causative of delay and which related to the parties’ contractual arrangements, concerned ICL’s original assessment of the development work and resources required to deliver the system. What conclusions did you reach in that regard?
Peter Copping: Well, ICL told us themselves that they had seriously underestimated the amount of work required, despite quite a long and protracted selection process as I understood it and, again, I think that was partly because of the complexity of the system. I think at this stage no-one really fully understood the implications of the end-to-end arrangements and the necessary interfaces to all of the systems outside of the Horizon project that needed to interface in order to make it all work and I think that, together with the agreements to agree issue, was a key cause for the delays.
Ms Hodge: I’d like to turn now to the findings you made at this stage about the programme’s management capability. In your report you express concerns about the resourcing of the programme and in particular about the level of managerial expertise within Post Office Counters; is that fair?
Peter Copping: Yes.
Ms Hodge: Would it be fair to say you also expressed some criticism of the programme delivery authority?
Peter Copping: Yes.
Ms Hodge: In your report you describe the PDA as focusing almost exclusively on achieving a high quality outcome, even potentially at the expense of timeliness and cost effectiveness.
Peter Copping: Yes.
Ms Hodge: That was one of the concerns you had at the time?
Peter Copping: Yes.
Ms Hodge: Was it your perception that a more pragmatic approach needed to be adopted, with trade-off’s being made between the performance of the product on the one hand and the business impact of delays –
Peter Copping: I think that’s fair, yes.
Ms Hodge: You ultimately concluded that there was no sensible way of de-scoping or radically altering the plan and that it was better to continue than to terminate; is that correct?
Peter Copping: Yes.
Ms Hodge: Did you understand the parties to be contemplating termination at this stage?
Peter Copping: There were veiled implications of termination, yes, at that stage for default against the contract by ICL, as I recall it.
Ms Hodge: On 24 September ‘97 you presented a summary of your findings to the PDA board –
Peter Copping: Yes.
Ms Hodge: – which they accepted, I believe –
Peter Copping: Yes.
Ms Hodge: – and the recommendations which you have made to minimise further delays to the programme. I think that was followed up by a meeting, a special meeting, of the PDA board on 2 October ‘97; do you recall that –
Peter Copping: Yes.
Ms Hodge: – where it was agreed that you would lead a series of workshops to examine the strategic risks to the programme?
Peter Copping: That’s right.
Ms Hodge: As well as the root causes of delay.
Peter Copping: Yes, that was essentially to ensure that there was a common agreement on the issues before the parties decided to proceed and investigate what they needed to do.
Ms Hodge: Did you regard those workshops as a success?
Peter Copping: They were, yes.
Ms Hodge: If I can move on now to the next significant stage of your involvement which came in the spring of 1998, you became involved in the programme again, on this occasion, at the behest of Her Majesty’s Treasury; is that right?
Peter Copping: Yes.
Ms Hodge: You were appointed to act as a consultant to an expert panel, chaired by Adrian Montague, the head of the Treasury private task-force on private finance. The panel, as I understand it, had been established to review the deliverability of the Horizon project together with the risks associated with the estimated timescales and cost of the programme. Is that a fair characterisation of their function?
Peter Copping: Yes.
Ms Hodge: So this is spring ‘98. Do you recall the extent of progress which had been made on the programme by that stage?
Peter Copping: No.
Ms Hodge: I think you might be assisted if I were to refer you to the written report which was produced by the expert panel at the conclusion of their review. I think it’s correct that you weren’t the author of that report but your findings contributed or were taken into account when that report was written; is that right?
Peter Copping: I didn’t know that there had been a report written at the time but I have now seen it and it certainly concurs with my understanding of what the Panel was going to say at the conclusion of our work.
Ms Hodge: Please could we pull up POL00028094. This is a copy of the report. We can see the composition of the panel there, Adrian Montague, Bill Robins and Alec Wylie, with whom you collaborated in the review.
Peter Copping: Yes.
Ms Hodge: Can we please turn to page 11. At point A, at the top of the page, there’s a heading “Current Status of the Programme”. Paragraph 21 reads:
“The programme has moved on since PA reviewed it towards the end of 1997”, and cites the following indicators of progress:
“Release 1c … a partial solution providing the benefit payment card and [order book control service] has been working satisfactorily in just over 200 offices since November 1997. Pathway has brought in new technical skills and management resources, increasing headcount to around 270 staff and introduced new procedures to support the high level of software development needed;
“BA has increased its resources on the programme and Release 3.0 of its key feeder systems (CAPS) has been given DSS [the Department of Social Security] Seals of Approval;
“[Post Office Counters Limited] has also increased is resources on the programme, establishing a pilot service management function and a National Implementation organisation, to support Pathway in preparing outlets and training; the Horizon Programme Office [referred to as the HPO] also started work on 1 April 1998.”
Does that reasonably encapsulate where things had got to by spring of ‘98?
Peter Copping: It’s a good reminder.
Ms Hodge: You were instructed by the panel to undertake a number of investigations into issues which you’ve identified in your statement. These were the extent to which Horizon was future proofed; whether it had the capability to support the Electronic Point of Sales Service; whether it could be developed to support simple banking applications; the likely lifetime of the technology; and whether the technology was suitable for long-term government infrastructure. Does that encapsulate the areas?
Peter Copping: That’s correct.
Ms Hodge: You have explained that, in carrying out these investigations, you held a series of meetings and one to one discussions with ICL –
Peter Copping: Yes.
Ms Hodge: – sorry, with ICL Pathway, as it was, and had extensive engagement with each of the parties over several weeks.
Peter Copping: Correct.
Ms Hodge: On this occasion, did you carry out any extensive analysis of the underlying documents?
Peter Copping: No.
Ms Hodge: Does it follow that your findings then were based, to a very great extent, on the information that you were given by the participants?
Peter Copping: Yes, that’s correct, together with external research that we would have done as to the state of play in the deployment of EPOSS, simple banking, and so on, and the comparison of that with what the system was capable of doing. If I recall correctly, we said we felt that the system could be developed to support those applications; the only question was how long it was going to take.
Ms Hodge: So if I can just clarify, I what I understand you to have explained is that, essentially, you weren’t looking at the parties’ underlying documents, you weren’t carrying out analysis of design documentation, and so forth, but you were carrying out some external research, in order to, I suppose, analyse what you were being told?
Peter Copping: Yes, we were calibrating the art of the possible against what we were being told by ICL Pathway.
Ms Hodge: To what extent did your discussions with ICL Pathway touch upon problems that had arisen during the development of the EPOSS application?
Peter Copping: Not at all. In fact, I don’t think the Montague review looked at issues at that time, problems, at all.
Ms Hodge: In your statement you say you have no recollection of being asked to enquire about the parties’ knowledge of technical faults and defects in Horizon during the review; is that right?
Peter Copping: Correct.
Ms Hodge: Albeit you weren’t asked, did you yourself make any enquiries into those matters?
Peter Copping: No.
Ms Hodge: Did you not consider the existence of known software problems might be relevant to Pathway’s capability to deliver the programme?
Peter Copping: No. We were looking at – well, as I understood it, the panel was looking at the possibility to reconstruct the programme in a way that would make it more deliverable and remove some of the risks that weren’t associated with the technical issues, on the basis that, if the programme could be reconstructed to achieve that, it was then a matter of time for ICL to deliver.
Ms Hodge: But if there were very serious technical issues, would that not be very relevant to whether they would ultimately be able to deliver?
Peter Copping: I guess it’s fair to say it could have been relevant.
Ms Hodge: In your statement, you explain that your overall view of the Horizon technology was positive –
Peter Copping: Yes.
Ms Hodge: – and that you believe the assessment you made at the time was accurate.
Peter Copping: Yes.
Ms Hodge: Would it be fair to say, though, that whether or not your assessment was accurate would depend on whether you were asking the right questions and you were receiving accurate answers to those questions?
Peter Copping: Well, inevitably, but the work that we did was guided by the panel and we were not asked to investigate any technical issues at that time.
Ms Hodge: If you weren’t making enquiry into the parties’ knowledge of faults and defects, how could you or the panel make an accurate assessment of Pathway’s ability to deliver this solution?
Peter Copping: I don’t think – that’s a difficult question to answer.
Ms Hodge: The panel reached a number of conclusions which appear to have been informed by your own investigations and findings and I’d like to explore some of those with you now, if I may. Could we pull back up POL00028094. Please could we turn to page 12. I think it might be internal page 12. Thank you.
Forgive me, I think maybe it’s page 11. It must be written down wrong. Could we go back one page, please. Thank you.
We can see at the bottom of this page, heading B, “Solution Design and Fitness for Purpose”, and if we could go to the following page please, page 11, at paragraph 25 this records that:
“The main architectural issues are scalability and robustness. We are advised [the panel said] that a solution of this scale and scope with so many different platforms and products has, as far as PA is aware [PA Consulting, I assume], no precedent. We are satisfied that Pathway’s approach to design, development and performance testing is sufficiently rigorous for such a major undertaking.”
Was that a reflection of the finding that you had made that end of your review?
Peter Copping: I think it’s probably more correct to say that was the panel’s conclusion. The lead technical person on the panel was an MOD man, Bill Robins and I think he probably did more than we did, in terms of investigating the technical aspects.
I think it’s probably fair to say and clarify we weren’t a member of the panel. We were there being asked to investigate specific aspects, I would imagine, in order to clarify their own view or to concur with their own view. In other words, we weren’t given free rein.
Ms Hodge: No, your role was to assist, essentially, and to follow the specific investigations that you were asked to follow.
Peter Copping: Yes. Yes, it was quite a different arrangement to the first review.
Ms Hodge: At paragraph 29, so on the same page we can see just over halfway down – thank you:
“The project is probably the biggest of its kind and many of the component parts, although sourced from industry strength products and companies, are being used towards their current limits and scale. Pathway has recognised the risks and has in place the controls we would expect to see in a development project of this scale.”
Again, was that based on your own findings or, as you recollect, was that a conclusion which the panel reached?
Peter Copping: I think that’s a conclusion the panel reached.
Ms Hodge: Turning then to future proofing, which was one of the aspects you were asked specifically to look at, at page 13, please, of POL00028094. At paragraph 33, so the very first paragraph, it records that:
“There is good evidence of future proofing at all levels. We have been satisfied that all reasonable steps have been taken to ensure robust sources of supply and compliance with industry standards in designing the architecture. Upgrades to software platforms and individual components are provided for, should they be necessary.”
Bearing in mind this was an aspect that you had looked at, as I understand, was that your finding or was this informed by your findings?
Peter Copping: That was our finding.
Ms Hodge: That was your finding. Insofar as you found that there had been compliance with industry standards in designing the architecture, that was a finding based on your external research of what those industry standards ought to be –
Peter Copping: Correct.
Ms Hodge: – and what you had been told by Pathway as to what they were doing?
Peter Copping: Yes, correct.
Ms Hodge: The report of the panel states that you carried out a critical path analysis to establish the risk of further delay to the programme; is that right?
Peter Copping: Yes.
Ms Hodge: One of the factors which you identified as being a likely cause of further delay was the absence of an agreement, any agreement, between the parties concerning the criteria and procedure for acceptance of the system; is that right?
Peter Copping: Correct.
Ms Hodge: What did you understand to be the areas of disagreement between the parties at this stage on the subject of acceptance?
Peter Copping: The disagreement was essentially about the specification and criteria for acceptance and, as I recall it, the basis on which acceptance would be given and whether acceptance should be on a model office or a live trial end-to-end under live conditions. That’s essentially what I recall.
Ms Hodge: To what extent had the earlier concerns that you had about the resourcing of the programme in the summer and autumn of ‘97 been addressed by the time of this review in July 1998?
Peter Copping: I think all parties had resourced up and my recollection is that there was still a concern that POCL were not ready to accept a system of this complexity, and that’s readiness in terms of preparing the network to live in a very highly structured environment, as opposed to a very unstructured environment at the branch using paper; a question about whether they were ready to receive a system in terms of the help facilities and helpdesk facilities that were outside of the technical helpdesk.
Ms Hodge: Did you have ongoing concerns about the competence or the expertise of the staff managing the post office counters aspect of the project?
Peter Copping: I think there were concerns in terms of the number of people involved and their technical competencies and their understanding of business process transformation that would be necessary to accept the system.
Ms Hodge: We know from the report that we’ve seen produced by the expert panel that one of the proposals which they made was the appointment of a neutral trouble shooter to facilitate negotiations between the parties over the future of the project?
Peter Copping: Yes.
Ms Hodge: Is that your recollection?
Peter Copping: Yes.
Ms Hodge: The individual appointed to carry out that role was Graham Corbett, Deputy Chairman of the Monopolies and Mergers Commission; is that correct?
Peter Copping: Yes.
Ms Hodge: You explain in your statement that Mr Corbett was tasked with advising ministers – that is ministers in government – on whether the framework suggested by the Treasury task-force would provide a commercial basis for continuing and whether the parties could develop a robust implementation plan to complete the project. Is that your recollection?
Peter Copping: Yes.
Ms Hodge: Because albeit, as you say, you didn’t author the report, you were aware that the expert panel had made a number of recommendations at the conclusion of their report; is that right?
Peter Copping: Yes.
Ms Hodge: That they were not in favour of terminating the project at that stage; is that right?
Peter Copping: That was my understanding.
Ms Hodge: What they proposed had been either a full restructuring or partial restructuring of the programme?
Peter Copping: Yes, yes.
Ms Hodge: So you became involved again in October 1998; is that right?
Peter Copping: Yes.
Ms Hodge: And at this stage, to test this feasibility of that restructuring exercise?
Peter Copping: Feasibility in the sense of the programme itself.
Ms Hodge: You have explained that you were asked to join a working group established by Mr Corbett; is that right?
Peter Copping: Yes.
Ms Hodge: Which was chaired by the director of the Horizon programme office. Do you recall who that was?
Peter Copping: No.
Ms Hodge: Does David Miller sound like the correct –
Peter Copping: David Miller was chair of?
Ms Hodge: Sorry, the director of the Horizon programme?
Peter Copping: Oh, yes of course, yes.
Ms Hodge: The purpose of the working group – sorry, your task was to liaise with the parties as they answered questions posed by Mr Corbett and to provide an assessment of any risks arising from the reconstruction; is that right?
Peter Copping: Yes.
Ms Hodge: We can see an agenda for one of those workshops at POL00090010, please. This is obviously a fax header sheet from you, Peter Copping to Peter Crahan, who was a senior figure in the Benefits Agency and Mr David Miller at the Horizon programme office and Mr Mike Coombs at ICL Pathway. On the following page, please, your letter confirming that you’ve made some proposals for the workshop and on the third page, please, we have here a list of a number of issues for resolution. The first of these is E2E and model office testing. That’s end-to-end; is that right?
Peter Copping: Correct.
Ms Hodge: And model office testing. You were looking for the Benefits Agency to explain the current concerns about the testing philosophy; is that right?
Peter Copping: Yes.
Ms Hodge: Do you recall what the nature of their concerns were at this stage?
Peter Copping: No.
Ms Hodge: You were looking to the Horizon programme office to describe the current process as well as the proposals for management of reporting of progress to sponsors, and it was your function, I believe we can see, PA to identify any remaining disagreements, issues, concerns and sensitivities on that subject.
Peter Copping: Yes.
Ms Hodge: So we also see, under topics for discussion, the service management product set. What does that mean?
Peter Copping: This would have been the construct around service management, who did what, where they would reside and so on.
Ms Hodge: The next section is the service management environment and the interim arrangements in place. Can you describe what those were, please.
Peter Copping: I really don’t recollect that level of detail I’m afraid.
Ms Hodge: At D we have multi benefit with soft EVP, that’s a reference to the security, the extended verification procedure; is that right?
Peter Copping: Yes.
Ms Hodge: I think you are seeking there for Pathway to describe the plans to realise NR2+. Do you recall, in terms of the future software development, what the plans were at this stage in relation to the new release?
Peter Copping: No.
Ms Hodge: It’s called NR2+ …
Peter Copping: I recognise NR2 but I couldn’t tell you what was in it.
Ms Hodge: Thank you. Then I think there’s one more page beginning E if we could zoom in. Thank you very much. Consistent and complete technical design, “BA to list areas where assurance is needed”. What were the BA’s concerns at this stage about the consistency and completeness of Pathway’s technical design?
Peter Copping: I’m sorry but I do not recollect.
Ms Hodge: We can see then that the final topics were acceptance testing and release authorisation and the Horizon programme office. Then, under item 2, the programme critical path and dependencies. Is this an accurate reflection of the types of issues that you were dealing, in the workshops that you were having –
Peter Copping: Yes, this is essentially a process that we will have taken the responses through, in order to flesh out areas of disagreement which would then be documented for someone to go away and work on and decide how to take those forward.
Ms Hodge: Your assessment of the programme and project management issues, which were prominent in autumn 1998, are summarised in an annex to Graham Corbett’s report; is that right?
Peter Copping: Yes.
Ms Hodge: Can we please show POL00028098, could we scroll down to page 32 please. We can see here at the top, “Management Summary – Key Programme Risks”. Is this a table that you produced or is it simply summarising your findings?
Peter Copping: I think it’s summarising our findings.
Ms Hodge: So in terms of the risks that you had identified, the first of those we see under the heading “Critical” is the speed of acceptance process. Can you explain, please, the nature of your concern at that stage about the speed of the acceptance process?
Peter Copping: My recollection is that things on – acceptance process got clogged up over disagreements on what the criteria were and how those criteria should be – differences should be resolved. I really can’t recollect any more than that.
Ms Hodge: In terms of the impact that this was likely to have on the programme, it records that:
“[A] Failure to complete acceptance in planned timescales could cause one or more of the parties to resort to legal action and program could stop at end of ‘98 or before.”
So was the essence of the concern that unless the acceptance process could be agreed and implemented it was likely to lead to litigation?
Peter Copping: Correct.
Ms Hodge: What did you understand Pathway’s position to be on acceptance at this stage?
Peter Copping: I think they were seeking to base acceptance on a self-certification process and of course no-one on the sponsor side was particularly happy with that. They also, I recall, were seeking to have acceptance on a model office as opposed to end-to-end acceptance, ie in a live system.
Ms Hodge: I will come back to the point about the model office and end-to-end testing shortly but, before I do, just scroll down please to the page, I think it will be 33, where we see what were described as the minor risks recorded. I wonder if we could zoom in please. Thank you.
So point 4, risk number 4, under the heading “Minor” is the “Consistent and complete technical design for key products”. The assessed impact of that on the programme is that it’s likely to impact mainly on the speed of testing and the acceptance process.
Peter Copping: Yes.
Ms Hodge: Did you not consider that the consistency and completeness of the technical design was relevant to assurance of the quality of the programme?
Peter Copping: I think it amounts to the same thing, doesn’t it?
Ms Hodge: Well, your focus here is on speed of testing and acceptance?
Peter Copping: If the criteria for acceptance are all agreed and the system is submitted against those criteria and there are no issues, then speed will be fairly quick. I’m not sure I’m following your point.
Ms Hodge: I think my point is this, that you seem here to be attributing the significance of the completeness and consistency of the technical design, its overall significance, to the programme is its likely impact on testing and the acceptance process. What I’m saying is that does it not also have a function in ensuring the quality of the solution that’s being put in place?
Peter Copping: Yes.
Ms Hodge: On the fifth point, also a minor risk we have “scalability of Pathway design”. Scalability was something that had been regarded as quite a significant issue in your earlier reviews in 1997 and 1998. Why is it here characterised as a minor risk to the programme?
Peter Copping: Well, because, at that stage, I think there was beginning to be a better understanding of how the system would be rolled out and scaled up, where scalability relates to the number of offices connected. So it was seen as less of an issue at this stage of the development. But, of course, there could always be issues. If you move from 10,000 to 20,000 offices, there might – suddenly an issue might arise.
Ms Hodge: Were both of these issues, that is the consistency and completeness of the technical design and the scalability of it, were they both not factors that were likely to affect, ultimately, the robustness of the system?
Peter Copping: Not necessarily but possibly.
Ms Hodge: There’s one topic I would like to deal with, please, before we have a short break. This comes back to the question of acceptance and you’ve mentioned in your evidence that you understood Pathway’s position to be that they were looking for acceptance to take place at the end of model office testing, as opposed to a full end-to-end test. Shortly after the negotiations that were being facilitated by Mr Corbett concluded, you wrote to David Miller, the Horizon programme director, in order to set out some private thoughts you had about how the parties might break the through the potential impasse on acceptance; is that right?
Peter Copping: Yes.
Ms Hodge: I wonder please if we could pull up POL0009009. Thank you.
So we can see here your letter of 19 October 1998, addressed to Mr David Miller, the director of the Horizon Programme, and, as I’ve just said, your proposal to set out some of your private thoughts on the issue of acceptance. That letter enclosed a short paper entitled “Acceptance testing: a framework for developing a new paradigm”. We can see that on page 2, please.
You observe in that paper, under the heading “Problem definition” that:
“Sponsors and Pathway have agreed to de-risk the programme by decoupling card roll-out from NR2”, which I understand is New Release 2; is that correct?
Peter Copping: Yes.
Ms Hodge: “… and to base NRO …”
Is that national roll-out?
Peter Copping: Yes.
Ms Hodge: “… on child benefit and EPOSS only until NR2+ [this is New Release 2 Plus, further functionality] is available when multi-benefit roll-out starts. This new sequence raises a legitimate question whether an alternative acceptance process can be designed that protects the commercial objectives of the parties and which at the same time reflects the status of the revised programme at completion of [Model Office Testing] and at completion of Live Trial. Additionally, the acceptance process for any requirement to be delivered during NRO [National roll-out] would need to be included in any new approach.”
Your paper essentially proposed a new paradigm for acceptance and we can see the essence of that distilled, please, on page 3. At the end of the second paragraph you say:
“Simply put [following your new paradigm], sponsors would give up termination rights on acceptance following [Model Office Testing] in exchange for the option of being able to have more punitive SLAs …”
Is that service level agreements?
Peter Copping: Correct.
Ms Hodge: “… following the start of [National roll-out], should the system fail to meet acceptance criteria in Live Trial. Similar arrangements could be put in place for future releases of functionality/services.”
Would it be fair to say that, boiled down to its core, your proposal envisaged the sponsors forfeiting their right to reject the system, even if it failed to meet the criteria, which the sponsors deemed to be necessary for acceptance?
Peter Copping: Yes, and perhaps I should put this in context. The clues in this letter are it was a private thoughts letter and, in the last paragraph, “Next steps”, bluntly, this an unsolicited proposal for more work from PA, and it was rejected.
Ms Hodge: It was indeed. Viewed from the perspective of the sponsors, this is an approach which would have been fraught with risk, would it not?
Peter Copping: Possibly.
Ms Hodge: By their very nature, model office tests tended to be carried out under optimal circumstances; that’s right, isn’t it?
Peter Copping: Yes.
Ms Hodge: Because they didn’t accurately replicate the real-life environment in which the system would actually operate, these tests were very unlikely to identify the full breadth of usability and performance issues –
Peter Copping: That’s correct.
Ms Hodge: – which would only become apparent, ultimately, in live operation of the system –
Peter Copping: Yes.
Ms Hodge: – by which point, the termination rights would have been lost?
Peter Copping: Yes.
Ms Hodge: So there was a risk in adopting this approach that the sponsors might find themselves bound to accept and roll out a system that later didn’t prove to be fit for the purpose for which it was –
Peter Copping: Which is why it was rejected.
Ms Hodge: Bearing in mind those risks inherent in the approach, why did you consider this to be a suitable paradigm for acceptance?
Peter Copping: We were trying to be creative to find a way through the acceptance block.
Ms Hodge: Was this reflective of the pragmatism which you felt was earlier lacking in the programme and which had contributed to significant delays?
Peter Copping: I don’t think we saw it in the broadest light. We saw that as a possible opening of a discussion that could help solve the problem. It was a pragmatic approach.
Ms Hodge: Thank you.
Sir, that brings me to the end of that topic. I wonder if now would be a convenient time to take a short break?
Sir Wyn Williams: Yes.
Ms Hodge: We’re making good progress.
Sir Wyn Williams: Good. So what time shall we start again Ms Hodge?
Ms Hodge: Shall we resume at 3.10?
Sir Wyn Williams: Okay, fine. See you then.
(2.55 pm)
(A short break)
(3.09 pm)
Ms Hodge: Good afternoon, sir. Can you hear, and see me?
Sir Wyn Williams: Yes.
Ms Hodge: Thank you. We can hear you.
Mr Copping, picking up where we left off, which was the new paradigm for acceptance which you had proposed (which, as you say, did not find favour with sponsors), you’ve explained in your statement that you continue to have some involvement in the public sector negotiations over the future of Horizon in the early part of 1999; is that right?
Peter Copping: Yes.
Ms Hodge: But the last significant engagement which you had related to the acceptance of the system; is that correct?
Peter Copping: Yes.
Ms Hodge: Contrary to the proposal which you had made which envisaged acceptance at the end of model office testing, we know that what Post Office Counters and Pathway agreed upon cancellation of the Benefits Payment Card was that an operational live trial would take place; is that right?
Peter Copping: Yes.
Ms Hodge: By that stage, however, the thresholds for acceptance had changed. Were you aware of that?
Peter Copping: Yes.
Ms Hodge: Could we please show POL00028208. Thank you. This is a copy of “Schedule A11” to the codified agreement, dated 28 July 1999, concluded between ICL Pathway and Post Office Counters. I suspect you won’t have seen a copy of this contract at the time.
Peter Copping: Correct.
Ms Hodge: Have you read this document since?
Peter Copping: Yes.
Ms Hodge: We can see, if we turn to the second page, please, at paragraph 2.2 – so about a third of the way down the page – a reference to the thresholds for acceptance of the CSR. Are you aware of what the CSR was or what it signifies?
Peter Copping: I don’t recall what CSR stands for, I’m afraid.
Ms Hodge: I believe it’s the Core Systems Release –
Peter Copping: Right.
Ms Hodge: – the name given to the package of software tested during the operational trial and ultimately rolled out. It comprised the EPOSS, Electronic Point of Sales Service, the order book control service, which was still in operation, and the Automated Payment Service. Does that sound broadly correct, in terms of what you understood –
Peter Copping: That sounds familiar, yes.
Ms Hodge: What this provision provides, we can see it’s framed in the negative. It effectively says:
“The thresholds will not be met if in respect of CSR Acceptance there are:
“[First condition] one or more high severity deficiencies as categorised in paragraph 7.1(a) of this Schedule (‘category (a) faults’);
“[Alternatively] more than 20 category (b) faults;
“[Finally] more than 10 category (b) faults in respect of any one CSR Acceptance Specification.”
Is that broadly consistent with what you understood at the time to be the overarching criteria that the system had to meet in order to be accepted?
Peter Copping: Yes, that’s correct.
Ms Hodge: So if there was one or more high severity deficiency, it wouldn’t be eligible for acceptance?
Peter Copping: Correct.
Ms Hodge: Likewise, if there were more than 20 of a medium severity, it wouldn’t be eligible?
Peter Copping: Correct.
Ms Hodge: We can see on page 3 at point 5, under the heading “Appointment of Expert”, that contract made provision for you to be appointed as an expert to assist in resolving any disputes relating to CSR acceptance. Is that how you understood your role at the time?
Peter Copping: I didn’t know about this at the time, so I didn’t understand I was being proposed.
Ms Hodge: When you did become involved, how would you characterise your role?
Peter Copping: It was explained to me that my role would be essentially to facilitate the parties to come to an agreement. There was a lot of allocation of blame from one party to another about what actually was going on on acceptance and it was explained to me that my role was to ensure the parties worked together to resolve conflict and, through that process, reach an agreement on the level of severity of each incident and a resolution plan.
I subsequently found out that, apparently, I had the opportunity to arbitrate but I don’t think, to the best of my recollection, that was ever exercised by either party. In other words, I had the option to tell them the way it was going to be on particular incidents.
Ms Hodge: Do you recall being consulted by either of the parties about the arrangements that were put in place in the contract?
Peter Copping: In this contract? No.
Ms Hodge: It appears, from your statement and from the records we’ve obtained, that you were first called upon to provide assistance to the parties on completion of the operational live trial. Is that consistent with your recollection?
Peter Copping: That resonates.
Ms Hodge: In preparation for a meeting which appears to have taken place on 16 August 1999, you were sent what was described as a “hot list” of Acceptance Incidents; is that right?
Peter Copping: Yes.
Ms Hodge: Please could we pull up POL00028355. This is an email from Tony Houghton, dated 13 August 1999. We can see you named as one of many recipients there, the fourth in the list. Can we turn to the following page, please. Here we have the “Acceptance Incident Hotlist”. Could you please explain what you understood the significance of this list to be?
Peter Copping: This was a list in which I identified all the incidents that were outstanding, together with those where there was a disagreement on severity and I think, from memory, there were three, possibly four, where POCL and ICL Pathway were in disagreement.
Ms Hodge: So I think, looking at the list, it appears as though there was disagreement in terms of severity on almost every single –
Peter Copping: Sorry, I was looking through the lens of the medium to high.
Ms Hodge: Forgive me. There were three incidents categorised by POCL as high. We can see the first of those in the list being Acceptance Incident number 376, which is described as the “Derived cash account not equal to the electronic cash account”.
The next high severity is in relation to training, number 218. It’s described as the “Training course Cash Account module inadequate”. POCL have assessed that as high, whereas Pathway were treating that as closed at this stage.
Then, thirdly, in the POCL infrastructure Acceptance Incident number 298, described as “Counter system subject to lockups & screen freezes requiring reboots”, assessed by Pathway as a low severity incident but by Post Office Counters as high.
I think there’s one in the category of medium to high, which was number 369, also in the POCL infrastructure, “Scanner reliability in relation to [order book control service] transactions”.
So that was the state of play at the end of the operational live trial, so far as you were aware; is that right?
Peter Copping: Yes.
Ms Hodge: Do you have any recollection of the meeting that took place on 16 August 1999?
Peter Copping: I might need reminding.
Ms Hodge: I don’t think we have any record, written record, of that meeting but we know that the disputes between ICL Pathway and Post Office Counters over the status and severity of these incidents were not resolved at that meeting. That much is clear from a supplemental agreement reached between the parties on 20 August 1999. I wonder if we could please pull up FUJ00000485.
So at the top we can see this described as a “supplemental agreement” dated 20 August 1999 between Post Office Counters and ICL Pathway. If we could please scroll to page 3 – thank you – under the heading “It is Agreed as follows”, we can see in relation to “CSR Acceptance”, paragraph 1.1:
“The parties agree that CSR Acceptance was not achieved as at the end of the CSR Operational Trial Review Period.”
If we go on, please, to page 4, there’s proposed here a remedy for the outstanding faults in the system. It provides:
“With a view to facilitating the obtaining of CSR Acceptance in the Second CSR Acceptance Test, the parties agree as follows …”
Paragraph 2.1:
“In the period between the date of this Agreement and 17th September 1999 (the ‘Limited Trial Period’), the parties will set up and conduct a programme of joint workshops for the purposes of agreeing (to the extent not already agreed):
“resolution plans for the Agreed Category B Faults, the Disputed Category A Faults, the Disputed Category B Faults … and (if appropriate) the Unagreed Fault …”
It goes on to say:
“a single timetable for resolution of outstanding category (b) faults”, would form part of those workshops.
I think you recall participating in those workshops; is that right?
Peter Copping: Yes.
Ms Hodge: We can see they were chaired by Keith Baines, the late Keith Baines, of Post Office Counters Limited and Tony Oppenheim of ICL Pathway?
Peter Copping: Yes.
Ms Hodge: On page 5 you’re referenced again. On this occasion, were you consulted about your involvement? I assume you were on the 16th.
Peter Copping: Not that I recall.
Ms Hodge: So here at 2.4:
“The parties will involve Peter Copping as Expert in the activities referred to in this paragraph 2 [to which we’ve just referred]. There may be occasions on which the Expert is asked to determine an issue as between parties in accordance with the provisions of Schedule A11 … but otherwise … his role will be as facilitator and adviser to the parties in their efforts to achieve successful resolution of outstanding issues.”
That seems to tally with what you have described earlier as your understanding, that is to say the latter part, facilitating agreement.
Peter Copping: I hadn’t seen this document at the time.
Ms Hodge: But it reflects, as far as you’re –
Peter Copping: It reflects my understanding.
Ms Hodge: You explained in your statement that, prior to each of the joint workshops to which this supplementary agreement makes reference, you received briefings from each of the parties relating to the status of the acceptance incidents; is that correct?
Peter Copping: Yes. That’s something I asked for, if I remember.
Ms Hodge: Did the briefings you received include ICL Pathway’s proposed plans for resolving the outstanding acceptance incidents?
Peter Copping: I’m not sure whether all the proposed plans were included.
Ms Hodge: We’ll take a look at some of them shortly. In the hot list that we reviewed a short time ago we looked at three acceptance incidents that were graded high by Post Office Counters. I would like to explore with you now what you recall about those.
If I could begin with Acceptance Incident 218, what did you understand this particular incident to entail?
Peter Copping: This concerned a post office view that the training for Horizon users was inadequate and I think, in response, Pathway offered a number of initiatives, which involved half day training for Post Office Counters staff in IT usage, computing, and so on.
The underlying issue here, I think, was to do with the situation that I mentioned earlier, that Pathway hadn’t, in completeness, considered the business transformation that would be necessary to accept the system and, with that, all the process changes that would need to take place at the counter, and my recollection is that there were extremes of counter configuration from quite a number of counters to a simple remote terminal in an outlying area, and users, if I recall correctly, between 20 and 75 or 85 years old. So there’s a tremendous spread of capability that needed to be trained.
In my understanding, that was why the Post Office decided this should be categorised as high severity impact.
Ms Hodge: Do you recall whether the concerns about training were focused on any particular aspect of the training programme being offered?
Peter Copping: My recollection is that the there were a lot of issues around closing of accounts and it was unclear, at that stage, whether that was a training issue or a systems issue.
Ms Hodge: So far as you’re aware, this was an incident that was resolved to the satisfaction of the Post Office during your workshops; is that right?
Peter Copping: Yes.
Ms Hodge: I’d like to turn to another incident, this was AI298, also categorised by the Post Office as being of high severity. Could you describe your understanding of that incident, please?
Peter Copping: This was about instability in the system, the symptom being lockouts at the terminal, crashes in the middle of a process, system busy incidents, and so on, and I think Pathway took the view that this was pretty normal for IT: PCs crash, PCs lock up. The Post Office or POCL considered this to have a high impact on the business simply because, while the system was down, customers couldn’t be dealt with and, therefore, it had a high impact and so there was a disagreement about the severity. I think POCL classified it as high severity; ICL Pathway, I think, as medium.
Ms Hodge: Do you recall how this particular incident was resolved?
Peter Copping: My recollection is that it wasn’t resolved through the series of seven workshops. It was escalated to the management resolution meeting towards the end of probably August/early September.
Ms Hodge: Thank you. We’ll return to that final workshop – that final meeting, sorry, a little later. Before we do, I’d like to address with you the third incident categorised by Post Office as high severity, and that is incident AI376.
What was the nature of the problem that had been identified, so far as you were –
Peter Copping: As I understood this, it was about intermittent failures in the reconciliation process between the money in the till and what the system had recorded and, at the time, I think it was believed the root cause was about – or lack of integrity in transfers between Post Office and ICL Pathway systems.
Ms Hodge: Do you recall where in the system this problem had arisen?
Peter Copping: At the TIP interface, as I understood it.
Ms Hodge: Can you explain what you understood TIP to be?
Peter Copping: It’s the interface between the Post Office system that is responsible for counting and the ICL Pathway interface and the database that recorded the transactions in the system.
Ms Hodge: You have described the root cause as being a lack of integrity in the information passing.
Peter Copping: That was my understanding at the time.
Ms Hodge: Before we go to one of the documents I’d like to show you, do you recall how the problem had come to light, how it had been detected in the system?
Peter Copping: Not specifically, I’m afraid, no.
Ms Hodge: Please could we show POL00028332. This is another email of 13 August 1999 from Andrew Simpkins, addressed to you and to David Rees. Was that a colleague at PA Consulting?
Peter Copping: Correct.
Ms Hodge: It says:
“Peter/David
“Following the Management Resolution meeting yesterday I attach as agreed by [Post Office Counters] and Pathway the minutes of this meeting and a summary of the incidents that are in dispute.
“The minutes will give you an up-to-date position on the high priority incidents in particular.
“We propose that the meeting with yourselves does not start until 12.00. It will be in Gavrelle House room 7.”
If we could please turn to the minutes on the following page, so of course not a meeting that you yourself attended but minutes that were shown to you to bring you up to speed for the meeting, to which we’ve referred, on the 16th.
Peter Copping: Yes.
Ms Hodge: On page 3, please, we can see, just over halfway down the page, at point 3, the heading “Review of High Priority Incidents”, the first of these being Acceptance Incident 376. “JD”, who I believe was John Dicks, an employee of ICL Pathway, reported that:
“Pathway recognise that not all transactions had been harvested and sent to TIP. A provisional fix went in on 2nd August and this has worked satisfactorily so far with the effect that all records had been sent. A root cause analysis has been developed, identifying 8 contributory problems, and all but one has been diagnosed and tested in Pathway to date. Pathway cannot guarantee however that all problems have been trapped. They will need to see evidence from the fix of the 8 known problems, and will continue to monitor the problems for 8 [sic] months to be confident of its resolution.
“The provisional fix and the control procedures developed allow Pathway to identify any errors, to patch the file, and to notify TIP in advance. Since implementation there have been no errors to report and hence Pathway contend that this action taken to date and the result they have observed justify the downgrading of this incident.”
We know, of course, it wasn’t ultimately agreed that that incident be downgraded –
Peter Copping: Correct.
Ms Hodge: – hence why the workshops took place. So far as Post Office Counters were concerned, what did you understand their assessment of the business impact of this incident to be?
Peter Copping: An inability on a consistent basis to reconcile Horizon data with cash data.
Ms Hodge: Would it be fair to say it was an issue of fundamental concern to Post Office Counters?
Peter Copping: I would have said so, yes.
Ms Hodge: The principal purpose of Horizon being to perform an accounting function –
Peter Copping: Absolutely, yes.
Ms Hodge: – which would enable Post Office Counters to reconcile the transaction performed by its agents at the branch counter with its own records of cash and stock held, as well as the transactions performed on behalf of its clients.
Peter Copping: Correct.
Ms Hodge: If the system wasn’t producing accurate cash accounts, which appears to be what this incident was showing, did this not call into question its very fitness for purpose?
Peter Copping: Yes.
Ms Hodge: Do you recall how ICL Pathway proposed to rectify this problem?
Peter Copping: In broad terms, yes. There were a number of proposals in the resolution plan. I think the most worrying was that one of the fixes wouldn’t be implemented until the year 2000, either at the end of ‘99 or 2000, and it was that that caused the incident to be escalated to the management meeting at the end of the seven workshops, not just because the final fix wouldn’t be available until the end of the year but also because regression testing would therefore take place afterwards and, if there were further issues to be found, that would not be something that would rest easily with roll-out.
Ms Hodge: If we could take a closer look at what was discussed at these workshops, you attended one on 26 August 1999; is that correct?
Peter Copping: If my name’s on the agenda, yes.
Ms Hodge: Could we show POL00028472, please. This is an email from Altea Walker to Graeme Seedall and others. You are not a recipient of the email but if we could please turn to page 6, we can see here a minute of the Acceptance Workshop – this is number 2 – held on 26 August 1999 and under the attendees you’re listed first, as the expert, Peter Copping, and the first item that appears to have been discussed is Acceptance Incident 376, the issue of data integrity.
At point 1, it records that:
“[Post Office Counters] needs to be confident of the root cause analysis and fixes, both applied and planned to be applied.”
The proposal was that a working group of Post Office Counters, comprising a number of employees there and Pathway, in brackets John Pope, an employee of ICL Pathway, were to review the TIP incident status report and report back progress and issues to the workshop.
Under point 3, we can see Pathway proposed to introduce a fix to ensure that the cash account does not lose transactions and there’s a reference to a PinICL. Were you aware what a PinICL was?
As I understand it, it was a record of an incident, the method by which Pathway recorded incidents in the system. As part of 1 above, that is to say the root cause analysis and fixes, a review was to be taken to fix and confirm acceptability of the fix to this group.
If we could turn the page, please, we can see at point 5 there’s a reference to Pathway proposing a three-level data integrity check to be implemented in December:
“This needs to be documented as a high level design including failure state analysis.”
We see there Post Office Counters Limited, a number of employees identified in brackets, are to be involved in interactive walk-throughs during the development of the design to report progress and issues to the group.
At point 6 it records that:
“[Post Office Counters’] position is that roll-out should not commence until data integrity can be assured. Ruth Holleran [an employee of Post Office Counters] to consider with the Auditors, and report back to this group, whether the current Pathway checks plus, possibly, continuing [Post Office] checks, would be adequate until Pathway’s full data integrity checks are in place.”
Finally, at point 8, we see a reference to Pathway preparing a rectification plan that will be presented to the group. So this appears to be the state of play as at 26 August 1999?
Peter Copping: Yes.
Ms Hodge: I understand you don’t have a detailed recollection of these events but does that broadly tally with what you understood the position to be?
Peter Copping: Yes.
Ms Hodge: Point 8, to which we have just referred, mentions a rectification plan that was produced in response to Acceptance Incident number 376. I believe you were shown a copy of that plan.
Peter Copping: Yes.
Ms Hodge: If we could bring that up, please, POL00028466. This document is dated 4 September 1999, we see that at the top, Version 0.3. Under the title it’s the “Acceptance Proposal for Acceptance Incident 376”, and the abstract records:
“This document contains ICL Pathway’s proposal to the independent Expert in respect of Acceptance Incident 376.”
Under distribution, you’re named as the first to receive it, “Expert: Peter Copping”. Do you recall seeing this document at the time?
Peter Copping: I do. I don’t know why they addressed it to me specifically.
Ms Hodge: If we could turn please to page 5, we can see here summarised Pathway’s position in relation to this incident. At paragraph 5.1, they set out the background:
“During the Live Trial and since, incidents have occurred that, in [Post Office Counters’] view, constitute a potential threat to the integrity of their accounts. These can be categorised into three groups.
“1. Some outlet transactions were not sent to TIP:
“because the harvester deliberately omitted incomplete records, caused principally by missing modes, and
“because, on one occasion, harvesting started before replication between recovering correspondence server nodes was complete.”
The second principal cause was that:
“Not all transactions were [completed] in the outlet cash account because of end-dating of Item Reference Data.”
Thirdly that:
“Some Cash Account records were sent to TIP because the pointer used by the harvester was not available:
“because a counter was rebooted before it could write it; and
“on one occasion, because a second balance process was allowed to run.”
Pathway suggested:
“Important advances have been made since the above incidents occurred, [which are] discussed below under the same numbers …”
We see here, I think, a list of fixes that have been applied to address the causes identified at 1, 2, 3 above, the first of these being that:
“All instances of messages written without harvester-sensitive fields have been fixed, except one that will be fixed shortly. Accounting integrity has been safeguarded by establishing routine examination of Event Logs to detect and report daily to TIP any harvester exceptions.
“The harvester has been enhanced to positively check that the full message set for an outlet is present on the correspondence server before initiating harvesting for that outlet.”
Secondly:
“The system is being modified so that the balancing and Cash Account processes can continue … if an item is end-dated during a period for which there are transactions.”
Thirdly:
“The system has been made robust against inopportune reboots by writing persistent objects to the message store, enabling controlled restart of the office balance process after power failure, etc.
“A change has been made to ensure that multiple balance processes cannot run concurrently. In addition a message will be displayed to inform the user that the balance process has initiated.”
So here we see Pathway essentially presenting a picture of three principal causes for the end balances having been identified. Would that be a fair characterisation?
Peter Copping: Yes.
Ms Hodge: When I say “root causes”, three overarching root causes and the fixes that they have applied?
Peter Copping: Or planned to apply.
Ms Hodge: Or planned to apply to address them. Do you recall how widespread these cash account discrepancies were at this time.
Peter Copping: In terms of quantity? No.
Ms Hodge: Could we please turn to page 8 of this document. It appears that this table was appended to ICL Pathway acceptance’s proposal. Do you recall being shown a copy of it at the time?
Peter Copping: I’ve certainly seen that before, yes.
Ms Hodge: Could we zoom in a little bit, please. Thank you.
So we can see at the top it is entitled “Incident analysis”. At the very bottom it confirms that these are the figures recorded as at 5 pm on Friday, 3 September, presumably 1999, bearing in mind the date of the document. At the very top we see “Number of outlets affected by cash account week”.
The top row appears to record the cash account weeks numbered 8 to 27, and the left-hand side column, the root causes. Do you agree with that broad analysis of the table?
Peter Copping: Yes.
Ms Hodge: So I think what we can see here is that there are two root causes, number 9 and 10 – so missing mode scales, replication recovery – that in the weeks 16 to 19 have caused quite a substantial number of outlets to be affected. In relation to 9, there are 22; in relation to 10 there are 37. Following the application of a fix, what this appears to record is that no further outlets have been affected.
But, in total over that period, we see 80 outlets affected by one of the 12 root causes of this problem; is that a fair picture?
Peter Copping: Yes.
Ms Hodge: It’s not necessarily the case that this equates to 80 different outlets, I think, because one possible interpretation of the table is that a single outlet was affected in more than one week but this is quite a high figure, is it not?
Peter Copping: Absolutely, yes.
Ms Hodge: Particularly when we bear in mind the relatively small number of outlets that were, in fact, operating the system at the time; is that right?
Peter Copping: Yes.
Ms Hodge: Do you recall how many branches were operating Horizon at this stage?
Peter Copping: At this stage, would it be about 200?
Ms Hodge: I think it was approximately 300, in the region of 299. When viewed in that context, 80 branches being affected by cash account discrepancies is very significant indeed.
Peter Copping: Absolutely.
Ms Hodge: We can also see from this table – if we can zoom back in, please – in the penultimate row it starts with a question mark “under investigation”, which appears to suggest that there were 36 branches affected by cash account discrepancies for which no root cause had been identified.
Peter Copping: Correct.
Ms Hodge: So, at this stage in early September 1999, would it be fair to say cash account balances remained a very serious problem?
Peter Copping: Yes.
Ms Hodge: In addition to the fixes that Pathway had identified in their resolution plan, they had also proposed introducing what was called a three-level data integrity check. We can see reference to that at paragraph 5.2 on page 6 of this document, please. At 5.2, the heading “Maturity of plan”, says:
“The Pathway proposal in this area has now been expanded into the High Level Design document Logical Design for EPOSS/TIP Reconciliation Controls.”
It goes on to review that was a document being reviewed by the working group in detail. Do you recall seeing a copy of that document?
Peter Copping: I think I’ve seen it somewhere. It might have been just recently, it might have been quite a long time ago.
Ms Hodge: Before we go to it, can you explain what you understood this three level data integrity check to entail?
Peter Copping: I thought you might ask me that and the answer is no. I’ve seen a very complicated description from ICL Pathway in a letter written by Tony Oppenheim, I think, in the pack.
Ms Hodge: If we could turn up, please, POL00090428. This contains a copy of the second supplemental agreement concluded between ICL Pathway and Post Office Counters on 24 September 1999, so at or around the time that your involvement came to an end. I think that’s right. If we could turn, please, to page 135 this is a copy of the “Logical design for EPOSS and TIP Reconciliation Controls”. It’s dated 20 September 1999, Version 0.7. It’s quite a lengthy document, I don’t intend to take you through all of its detail but if we could turn, please, to page 6 – I apologise, it was 135. It’s internal page 6. Forgive me, so that should be 141.
Under the heading 3, “Overview”, there’s quite a helpful summary of what the process was intended to entail. It records:
“The reconciliation processes will be split into two separate sets of activity. Daily reconciliation tasks and Weekly (or more accurately at the end of each [Cash Accounting Period] CAP) reconciliation tasks.
“The daily tasks will ensure that the base transaction data recorded at the counter matches the base transaction data transferred to TIP for that day. At the same time, the transactions will be used to generate total control totals for the Cash Account tables to which the transactions will report at the end of the [Cash Account Period].
“At the end of the [Cash Account Period], the daily control totals generated for each Cash Account table will be accumulated and the resulting value calculated for the Payments and Receipts table will be compared with the Cash Account line records generated by the Cash Account production process. If there is a discrepancy in this comparison, then the system will validate each of the accumulated daily control totals with the corresponding Cash Account line records to identify the table which does not reconcile and record an error message in the Riposte message store.
“The existing functions in the system which create the outlet stock holding records and the Cash Account Line records will also be amended to accumulate a control total for each set of records which will be written into the message store at the end of each set. These control total records will be harvested and inserted into the TPS Host database. The TPS Host system will compare the Stock Holding records … and the Cash Account Line records … output to the TIP Cash Account subfile with the control totals received from the OPS system. In the event that the TPS harvester fails to locate either the Stock Holding … records or the Cash Account Line records … or the control records calculated by the TPS Host system differ from the control totals received from the OPS, then a reconciliation error report will be produced.”
Does that assist you at all in relation to how this piece of software was intended to function?
Peter Copping: It resonates and of course the big question is what happens with the error report data?
Ms Hodge: If I’ve understood it correctly – and that is a big if – these new reconciliation controls, I think, were intended to automate a task that Post Office Counters had been performing temporarily in TIP – is that right – in that they had been seeking to verify whether the base transaction data recorded at the counter was consistent with the transaction data being transferred to TIP?
Peter Copping: Yes, I think the aim was to have a completely automatic reconciliation process, which corrected, as a result of the process the checking process. Whether that happened in practice, I can’t say.
Ms Hodge: I think, as you have said, it was a system which was designed to generate a report to verify that the error had taken place –
Peter Copping: Mmm.
Ms Hodge: – forgive me, to verify that in imbalance had been detected but what the reconciliation control didn’t do, did it, was identify the root cause of the discrepancy in the first place?
Peter Copping: I think that’s right.
Ms Hodge: As you have explained, you attended a series of workshops, the last of which, I believe, took place on 17 September 1999; is that right?
Peter Copping: I think so.
Ms Hodge: Do you recall what progress had been made in relation to resolving IA376 by that stage?
Peter Copping: It was still, as I recall it, categorised as high by POCL and medium by ICL.
Ms Hodge: Please could FUJ00079716 be shown on the screen?
We can see your name’s recorded under attendees at this meeting on Friday, 17 September 1999. This was the last of the seven workshops that had been arranged.
Peter Copping: Correct.
Ms Hodge: If we could turn to page 6, please, so at the bottom is the heading “AI376 Data Integrity”, the substance of which we can see on the following page. If we could scroll down, please. So here is, essentially, an update on where things are in relation to the data integrity checks. It records, as we’ve seen before, that Pathway were proposing a three-level data integrity check to be implemented in December, with the relevant design documentation to be considered.
At the fourth workshop, the update was that Post Office Counters had considered the high level design to be generally good but wanted further checks to be undertaken in relation to failure scenarios and operating procedures.
We can see then under workshop number 5 that Post Office Counters had reviewed identified failure scenarios and some issues with the high level design, which needed clarification. It was proposed that a meeting would take place to discuss Post Office Counters’ paper, the EPOSS/TIP reconciliation controls, summarising the failure scenarios and the design issues.
A further update at the sixth workshop was that progress was good and that Pathway were to issue a paper for Post Office Counters to review.
The final update is at the bottom there for workshop 7. Pathway had issued the high level design – I assume HLD means “high level design” – paper for Post Office Counters’ review and Post Office Counters to provide their written comments to John Pope.
So that’s essentially where we were with the high level design. Could we go to the next page, please.
In relation to whether or not to accept and roll-out out the system, this confirms Pathway’s position, as stated previously, was that roll-out should not commence until data integrity should be assured. At workshop number 4, the update was to the effect that Post Office Counters and Pathway needed to develop a contingent approach, possibly including indemnities, suggesting that Keith Baines and Tony Oppenheim would meet with the lawyers to initiate that process.
At workshop number 5, Post Office Counters’ position remained that the incident should be classified high until the data integrity fix was in place. Further internal meetings were proposed to further confirm the position.
Workshop number 6, “This will now be part of the contractual discussions being held” between Post Office Counters and Pathway.
At workshop number 7, the one that was held on 17 September, it said:
“This issue is now focused on the success criteria for [national roll-out] resumption.”
It confirms at a review in November – which may be an error because, of course, these minutes are dated September – Pathway had previously proposed four weeks’ operation with a less than 1.5 error rate. Keith Baines and Ruth Holleran proposed an error rate of 0.6 per cent, the current average being 1.2 per cent, together with six other conditions, five of which were listed in a paper that “RH” – presumably Ruth Holleran – had produced, and the sixth being a further two-week period of live running of the permanent Cash Account fix, prior to the actual recommencement of national roll-out in January.
The penultimate paragraph records Tony Oppenheim responding as follows: a 0.6 error rate agreed subject to this being measured as the average of six weeks from 4 October to mid November, with a maximum of ten working days to analyse each TIP fault, comprising a root cause analysis, diagnosis and agreed resolution and that was agreed, except for faults requiring diagnostics. A further two-week period agreed, subject to the agreement of logistics of the plan. On the basis of the current plan, this condition would lead to a two week delay in the planned date for recommencement of national roll-out and this was agreed:
“Re the error rate criterion, the Cash Account does not reconcile and is attributable to an error in the POCL domain. The error rate is to be calculated as the ratio if the number of incidents and the total number of cash accounts during the six weeks period.”
So this is where we were on 17 September. What do you understand these discussions to relate to?
Peter Copping: Two things. There was still a problem with 376 and there was some negotiation beginning to start about what the acceptance criteria might be for that particular incident.
Ms Hodge: You use the term “acceptance criteria”. Would it be right to refer to it maybe as “conditions”, the conditions upon which the system might be accepted?
Peter Copping: Yes.
Ms Hodge: So what we have, effectively, here is evidence of discussions taking place concerning the conditions on which Post Office Counters might be prepared to accept the system, notwithstanding that ongoing cash account balances were being affected?
Peter Copping: Yes.
Ms Hodge: The proposal from Keith Baines and Ruth Holleran, employees of Post Office Counters, was that an error rate of 0.6 per cent, together with a number of other conditions, would be acceptable. Was that your understanding at the time?
Peter Copping: It resonates with me, yes.
Ms Hodge: Do you recall the advice that you gave to the parties concerning the conditions upon which the system might be accepted in late September 1999?
Peter Copping: I don’t recall giving advice on 376.
Ms Hodge: Were you in favour of Post Office granting conditional acceptance to the system at this stage?
Peter Copping: I don’t think I indicated that, no.
Ms Hodge: Are you essentially saying that you played no part in facilitating the resolution of this particular incident by this late stage in September?
Peter Copping: By implication, I suppose I had an impact in the parties getting to that position but there was a subsequent meeting between the two senior people, which continued to debate what those conditions should be and that ended in an agreement that further staff work was necessary to understand the implications of those agreements and I didn’t play a part in that. My understanding was that there was some further negotiation which resulted in an agreement for somewhat different conditions but I don’t know what they were.
Ms Hodge: Just dealing, first, with the level of your involvement with this AI, you were of course present at the meeting on the 17th, when these conditions were being mooted as a possible condition for accepting the system and rolling it out?
Peter Copping: Yes.
Ms Hodge: Even if you didn’t give specific advice on it, you were aware that this was what was under discussion by the parties; that’s right?
Peter Copping: Yes.
Ms Hodge: To accept the system with ongoing cash account imbalances, did you not consider at the time that this represented quite a significant risk to Post Office Counters?
Peter Copping: And I think we discussed that.
Ms Hodge: Because an error rate of 0.6 per cent in any given week would have equated to more than 100 Post Office branches when spread out to the national level.
Peter Copping: No, it’s quite significant.
Ms Hodge: Now, I think you say you don’t recall having any part in the final resolution that was reached. I wonder if we could look at POL00083907. This is an email from Keith Baines on 22 September 1999 to a number of employees of Post Office Counters, Andrew Simpkins, John Meagher, David Miller, David Smith and Ruth Holleran. It records – the subject of it is “The AB and RAB on Friday”. Do you know what that is a reference to?
Peter Copping: Acceptance Board and Release Acceptance Board.
Ms Hodge: I think it’s the “Acceptance Board” and the “Release Authorisation Board”, possibly?
Peter Copping: Right.
Ms Hodge: It says:
“John/Andrew
“At this morning’s briefing session with Stuart Sweetman on the acceptance position, there were some discussions about the role and empowerment of the AB and RAB. I have since spoken to Jeff Triggs and obtained his view on this and then discussed with David Miller. The position is as follows:
“Post Office Counters will not be accepting the service against the existing contract and therefore the nature of the decision at the acceptance board is different to that originally intended. The board should make a recommendation as to whether or not the second supplementary agreement which has been negotiated with Pathway over the last few weeks should be signed. The supplementary agreement then states that acceptance is deemed to have taken place and the various contractual consequences of that, such as payment to Pathway, will follow.
“The same applies to the RAB since the supplementary agreement says that Post Office Counters Limited has authorised roll-out. The supplementary agreement is formally a change control note to the contract and, therefore, can be signed by David Miller – it doesn’t need Stuart’s signature. Can you please make this rather subtle change in the roles of the meetings apparent in their agendas, please?”
That’s from Keith at 13.27.
Can we turn to the next page please. There’s a further email on the same date at 13.51, the subject being the “Supplementary Agreement”:
“The enclosed is my understanding of the position we agreed at the end of yesterday’s meeting with Pathway. There is one area not yet agreed – namely the question of how to count incidents under AI298. The wording in the enclosure is that suggested by Pathway and recommended by Peter Copping. We were not able to agree it yesterday because we don’t have the right people available to review it. Can John and Ruth look and comment to me? Copied to Jeff Triggs, please.”
So can we turn to the following page, please. So here we have the position reached in negotiations on the 21st, relating to Acceptance Incident 298. What the previous email suggests is you had some input on the wording of this particular –
Peter Copping: On 298 –
Ms Hodge: Do you recall that?
Peter Copping: – yes.
Ms Hodge: That provided that:
“The occurrence of operational incidents in connection with this AI should have been reduced below a target threshold as measured over the four weeks. Measurement will be based on all outlets installed before or on 1 October 1999, provided there are at least 750 such outlets.”
If we scroll down to the penultimate paragraph:
“The target to be met is that the rate of occurrence measured over the four-week period to mid-November should average no more than one unit per counter position per three months.”
So that essentially was the target set for resolution of AI 298; is that correct?
Peter Copping: Correct.
Ms Hodge: And on the following page, please, we have Acceptance Incident 376. This refers to the arrangements for the integrity control to be implemented by Pathway by 31 December 1999 and that those will be as previously required by Post Office Counters, apart from the following amendments listed below. So is it right to say your evidence is you had no involvement in the detail of –
Peter Copping: I think this was an agreement that was struck between POCL and Pathway without my involvement.
Ms Hodge: After the workshop on 17 September 1999, do you recall having any further involvement in the Horizon System?
Peter Copping: My involvement ceased after 24 September ‘99.
Ms Hodge: Forgive me, the meeting to which – the further resolution meeting to which we’ve referred?
Peter Copping: Sorry, I’m not following.
Ms Hodge: Sorry, my question was whether your involvement ended on the 17th but you attended a further meeting as we’ve just –
Peter Copping: I attended one meeting which was between Richard Christou and David Miller, I think, which was the first stage of the escalation process defined in the acceptance documentation. That meeting ended with an agreement between the two that further staff work would be necessary in order to understand whether or not there could be an agreement on 376. I did not take part in that process and I’m not familiar with the output.
Ms Hodge: At the point at which you ceased to be involved in Horizon, what was your professional assessment of the robustness of the system?
Peter Copping: I think my overall assessment was that the Post Office had, in accepting the system – and this is a benefit-of-hindsight judgement – had accepted further risk in agreeing to accept the system and release for roll-out with the proposals from ICL, particularly on 376. We certainly talked about what needed to be put in place in order to monitor and mitigate any risk arising, but I really don’t know what happened after I left the project in terms of risk mitigation and further testing of the bug fixes that were being put in place beyond the acceptance timescale.
Ms Hodge: Thank you, Mr Copping. I’ve no further questions for you. There may be some questions from the representatives of the Core Participants.
Questioned by Mr Stein
Mr Stein: Sir, yes, I have just one short area of questions that have been permitted by the Inquiry team.
Mr Copping, my name is Sam Stein. I represent a number of – a large group of subpostmasters, mistresses and managers.
I’m going to take you to your report that you dealt with already with my learned friend at POL00028092. Paul, if you’re handling – thank you very much – if you’re handling what we see on the screen, could you go to internal pagination on Relativity page 7 of 132 and roughly two-thirds of the way down you will see the paragraph, Paul, that starts:
“Our key concern …”
Could you highlight that paragraph. Thank you.
Now, Mr Copping, I am just going to remind you of what is being said here within this report:
“Our key concern is that the skills required for many of the new senior post are, in our opinion, not those we would have expected to find as part of POCL core competencies. This is especially true in relation to implementation management and contract and service management. There seems, however, to be no evidence of external recruitment activity or robust plans to create the competence internally.”
Mr Copping, that seems to foreshadow a fairly bad problem within POCL; do you agree?
Peter Copping: Yes.
Mr Stein: The reason for that is that you are talking about major parts of the future planning. That’s implementation management; that’s putting it into place?
Peter Copping: Yes.
Mr Stein: Contract – that’s oversight presumably of the contract – to get Horizon working?
Peter Copping: Yes.
Mr Stein: And then oversight of the service which is then being provided by Horizon?
Peter Copping: Yes.
Mr Stein: These are core competencies?
Peter Copping: Yes.
Mr Stein: When subpostmasters and mistresses started to use the Horizon System, they found, as their evidence has set out, that the training was inadequate, that there were difficulties with the operation of the Horizon System, and, in particular, difficulties with trying to make sure that they could achieve balance.
Peter Copping: Yes.
Mr Stein: Achieving balance in relation to their accounts is an important part of their process; do you agree?
Peter Copping: Yes.
Mr Stein: If there is insufficient oversight and knowledge of the Horizon System within the Post Office, within POCL, does that mean that these particular difficulties that postmasters and mistresses were suffering from might not be remedied by the Post Office?
Peter Copping: I don’t think I can answer that question. I think at the time we did this review in ‘97 there was no doubt in our minds that the Post Office had a shortfall in competent resources in the areas we discussed. There is no doubt in my mind that the Post Office did resource up, and my recollection would be that David Miller was the first significant appointment that was made in that resourcing up process.
I would maintain at later stages of my involvement that the Post Office had a shortfall of what I’d describe as general technical competence that was capable of properly interrogating the Pathway personnel as to exactly what was going on in the development process and everything that flows from that.
Mr Stein: So, in other words, your concern then, and concern remains towards the end of your time working on this project, that the Post Office might not have the technical ability to understand what’s actually happening within the system; is that fair?
Peter Copping: I think that’s fair comment and it’s broader than that as well. There was also a readiness for acceptance of the system within the Post Office POCL organisation and all that implies in terms of service management on the Post Office side, as opposed to the technical side, which was ICL’s responsibility, and the need for process change in order to support new ways of working.
Mr Stein: And finally, it’s stating perhaps the blindingly obvious but Horizon was a new way of working for the Post Office?
Peter Copping: Absolutely.
Mr Stein: Thank you, sir.
Sir Wyn Williams: All right. Well, thank you very much, Mr Copping, for providing written evidence to the Inquiry and for coming to give oral evidence. I’m grateful to you.
Ms Hodge: Sir, I believe Ms Page has some questions for the witness as well.
Sir Wyn Williams: Oh, I see. Sorry, I hadn’t appreciated that.
Questioned by Ms Page
Ms Page: Just one, in fact, from the questions we were permitted and it’s on Acceptance Incident 376, which you’ve just described, as you did in your statement, as something which you understood POCL to be taking on more risk as a result of the position as it was left when you no longer were involved.
Peter Copping: Correct.
Ms Page: Would you therefore have expected those risks that they were taking on, those additional risks that they were taking on, to be registered in some way perhaps by the board or by some management level and monitored until they were satisfied that those risks were no longer significant?
Peter Copping: Yes, and I don’t know whether this was put in place. What would normally happen on a project of that sort would be a full risk assessment process which is updated on a regular basis within the programme, and if insufficient progress is being made (for example, on bug fixes and regression testing), then those risks would begin to replicate themselves over time and that, in turn, one would expect would escalate the issue to a higher level through the organisation. Whether or not that was put in place, I can’t say.
Ms Page: But that’s what you would have expected?
Peter Copping: That’s what I would expect from a management point of view, yes.
Ms Page: Thank you.
Sir Wyn Williams: Right. Well, I won’t repeat my thanks but thanks again, Mr Copping. And I take it that now is the end of this session?
Ms Hodge: Yes, sir. Thank you that concludes the evidence of today.
Sir Wyn Williams: All right. Well, we will meet again on Tuesday morning at 10.00. Thank you all very much.
Ms Hodge: Thank you.
(4.27 pm)
(Adjourned until 10.00 am on Tuesday, 25 October 2022)