17 November 2022 – Charles Cipione
Hide video Show video
(10.00 am)
Mr Beer: Good morning, sir. Can you see and hear me?
Sir Wyn Williams: Yes, thank you very much.
Mr Beer: Can I recall Charles Cipione, please.
Sir Wyn Williams: Yes.
Charles Cipione
CHARLES CIPIONE (sworn).
Questioned by Mr Beer
Mr Beer: Please take a seat, Mr Cipione. As we established on the last occasion, your report is 174 pages in length and is divided into two parts, parts 1 and 2. Part 1 is between pages 12 and 64, and part 2 is between pages 65 and 160. You’re giving evidence today, as you know, about part 2 of your report.
Can you take open part 2 of your report, please. It is at page 65 in the hard copy and on screen it’s EXPG0000001. Thank you.
That runs as I say between pages 65 and 160. Can you please confirm the following. Firstly, that you have made clear in part 2 of your report which facts and matters referred to are within your own knowledge and which are not?
Charles Cipione: Yes.
Mr Beer: Secondly, that those facts in part 2 of your report which are within your own knowledge you confirm to be true?
Charles Cipione: Yes.
Mr Beer: And thirdly that the opinions you have expressed in part 2 of your report represent your true and complete professional opinions on the matters to which they refer?
Charles Cipione: Yes.
Mr Beer: Thank you. Can we turn to the next page, please which is your methodology. In this section of your report and over the next six pages you address the methodology that you and your team adopted to analyse the primary material that you were provided with; is that right?
Charles Cipione: That is correct.
Mr Beer: Now, rather than going through each of the paragraphs on those six pages, I would be grateful if you would describe in summary terms the process that you and your team adopted to the very large cohort of material that you were given.
Charles Cipione: Absolutely, absolutely. So we’ll take this section by section. The first section 7.2 is the analytics work stream. What I am describing here is the fact that we wanted to try to structure the information in the PEAKs and PinICLs in a way that we could then do further analysis on that.
Effectively, what we did was we took a number of HTML and PDF files and converted them into a more digitised form so that we could do searching and whatnot, and then feed it into further processes. So effectively that is what 7.2 is describing.
Mr Beer: So it’s the very first stage, getting the primary material into a more readable and searchable format?
Charles Cipione: Exactly.
Mr Beer: Yes. And after that analytics work stream, what was next?
Charles Cipione: So as further downstream process following the analytics work stream, we also wanted to incorporate a couple of technologies for the review purposes of the project. The two technologies are Relativity and Brainspace. Relativity is an E discovery platform for viewing documents. So we wanted to populate the information into Relativity, all of the PEAKs and PinICLs, as well as the KELs and monthly reports if we needed them in there for monthly purposes.
Mr Beer: Just to remind us from last time, I think you received 56,489 PinICLs and 16,530 PEAKs. Does that sound right?
Charles Cipione: That sounds right, yes.
Mr Beer: Thank you. Sorry, I interrupted.
Charles Cipione: So Relativity is just a tool to help for the review process, as simple as that. Brainspace is a tool to help organise the searching methodologies that we were using for some of the work that we were doing. Effectively it is a machine learning tool that allows us to group the information especially as it came from the analytics work section to help us narrow in and identify specific documents or groups of documents as needed throughout the review. So effectively we loaded the information into there, into Relativity and into Brainspace, as part of the process.
So we actually loaded in the HTML documents, but we associated them with the work that was performed in the analytics work stream. So they were paired up.
Mr Beer: Tied with each other?
Charles Cipione: Exactly. Perhaps it might be interesting. You know, some of the information that we were interested in seeing, especially in the PEAKs and the PinICLs, were bits of information out of the PEAKs and PinICLs referred to as response categories and defect causes. They were just a standard bit of information that were in the PEAKs and PinICLs.
So we – I do reference where we had our reference information from there and, if you look on figure 7.1, it shows how that would have been displayed in the Relativity system as the team was looking through their initial review to identify documents for me to review.
Mr Beer: Thank you. I think you also received 656 KELs; is that right?
Charles Cipione: That is correct. So we did a similar process with the KELs. We ran it through this analytics work stream and we made that information available in Relativity and Brainspace for review purposes.
Mr Beer: I think you were only able to examine a proportion of the KELs that you identified in the period that you were looking at, and the reason for that was that you were, I think, told that some KELs had been deleted. Do you know why the KELs had been deleted?
Charles Cipione: Let me make sure.
Mr Beer: It’s a footer 69, 7.4.4, 7.4.5.
Charles Cipione: Yes, my understanding was some of the KELs were deleted and weren’t available for me to receive for review.
Mr Beer: Did you, from the material that you did examine, understand why the KELs may have been deleted or not?
Charles Cipione: Other than they were deleted they could have timed out. Perhaps they weren’t relevant for retention from Fujitsu, but past that it would be speculation on my part.
Mr Beer: Thank you very much. You received a series of management reports, I think two, and you gave us the numbers of those on the last occasion and the different species of the management reports, monthly Pathway ICL reports and the like. They are set out in the previous section of your report in 6.4. I’m not going to go there.
You explained overall on the last occasion that what you described as a brute-force approach was not possible. Can you just remind us of what you meant by brute-force approach not being possible.
Charles Cipione: Right. So the brute-force approach was not a practical approach to review the PEAKs and PinICLs.
Mr Beer: Yes.
Charles Cipione: However, for the management reports I did review those. I reviewed all of those. So the review process for the management reports was – I read them. I read all of them, sometimes several times. For the PEAKs and the PinICLs it was more of the Relativity review process, and we can get into that a little bit more if you want as far as, you know, we would target particular words, we would have Brainspace help us identify themes that we were interested in looking in, and have Brainspace assist us in identifying the PEAKs or PinICLs that were relevant to the particular theme that we were investigating.
So PEAKs and PinICLs, brute force, too many documents, not enough time, and we did not do that. For the monthly reports and the KELs we did review all of those.
Mr Beer: Thank you very much.
In terms of the limitations of the exercise that you undertook, you took us through on the last occasion the limitations of it. Just to remind us of some of them, firstly, the material that you were examining came principally from Fujitsu rather than the Post Office.
Charles Cipione: That is correct.
Mr Beer: Therefore you don’t have, as it were, the Post Office side of the house, side of the story, insofar as concerns the issues that you speak to in your report.
Charles Cipione: That is correct. There are a few of the monthly reports that were jointly issued by the Post Office and by ICL Pathway, but the majority of the information that was used for the themes and opinions I arrive at come solely from ICL Pathway.
Mr Beer: On occasions, the ICL Pathway-issued monthly reports attribute a view to the Post Office, but that’s coming from ICL Pathway rather than the Post Office itself?
Charles Cipione: That is correct. To the extent that I cite anything that refers to the Post Office, unless it’s from the joint report, that is ICL Pathway’s view concerning the Post Office. It’s not the Post Office’s view.
Mr Beer: We’ll come to look at an example of the joint report where it’s co-authored by the Post Office and Fujitsu in a moment.
Secondly, in terms of the limitation of the time period of the exercise you were undertaking, it restricted to which years?
Charles Cipione: I believe it was 1996 through 2000 was the material that was covering that time period.
Mr Beer: Now, I believe you have been given access to the witness statement of the witnesses who have to date given evidence in Phase 2 of the Inquiry.
Charles Cipione: That is correct.
Mr Beer: You have been given access to the transcripts of witnesses who have to date given evidence in Phase 2 of the Inquiry?
Charles Cipione: Yes.
Mr Beer: Has anything in the witness statements that you have been given or the transcripts that you have been able to read given you cause to revise or change any of the opinions or the contents of your report in part 2?
Charles Cipione: I have not read anything in the witness statements or the transcripts that gives me pause concerning any of my opinions.
Mr Beer: Before we get into the detail – and we’re going to come back to this at the end, when looking at some of your overall conclusions that are expressed in the executive summary – at a high level, could you summarise it for us, the impact, if any, that the written and oral evidence that you have been able to read had on the views expressed in your report. That is a very broad question.
Charles Cipione: Yes. Could you ask me the question one more time, please.
Mr Beer: Yes. Looking at the witness evidence that you have read, both the witness statements and the transcripts, can you summarise for us what impact, if any, that evidence had on the views expressed in your report.
Charles Cipione: As far as the views expressed in my report, I believe it confirms most of the views that I’ve expressed in my report. The witness statements and the transcript information that I’ve read so far seems to run in parallel with the opinions that I’ve expressed here.
Mr Beer: So would this be a fair way of putting it: it solidified the conclusions that you reached?
Charles Cipione: Yes.
Mr Beer: In the light of the witness statements and transcripts of the oral evidence that you’ve read, have you formed any view as to what you now know as to the use of some of the data that the Horizon System produced for criminal justice purposes, i.e. to seek to prove beyond reasonable doubt that a subpostmaster committed an offence of theft or false accounting?
Charles Cipione: Are you asking me if I have an opinion on whether there was a high level of integrity within the data that was used?
Mr Beer: Yes.
Charles Cipione: Yes. Based off of the information in the witness statements and the transcripts, I think that the integrity of the information used would be suspect.
Mr Beer: You told us on the last occasion that part of your background, a significant part of it, involved the design, creation and maintenance of systems that are used in legal proceedings.
Charles Cipione: That is correct.
Mr Beer: Can you just remind us in summary form what that was –
Charles Cipione: Yes.
Mr Beer: – or what that is.
Charles Cipione: What that is. So the firm I work for, AlixPartners, does a lot of work in the turnaround restructuring space in the United States. That is a legal proceeding that requires a lot of reporting to the court, and it requires basically presenting defensible data as far as the court is concerned. When I say defensible, what I mean is the data has to be complete and accurate and, to support that completeness and accuracy, it’s important for the people producing the data and any analyses, reports that are associated with that data to maintain a high level of transparency and auditability within those datasets.
Mr Beer: What do you mean by a high level of transparency and auditability?
Charles Cipione: What I mean is it’s apparent what happened with the data, and you can look at that two ways. When I say transparency, I mean you’re talking about all the processes, all the manipulations, it’s apparent through the coding or the queries or the reporting exactly what manipulations are happening there.
Mr Beer: And transparent to who?
Charles Cipione: Transparent to anyone that wants to look at it: transparent for the court, transparent for the debtors, transparent for the creditors, which are the two sides in the Bankruptcy Court in the United States.
Mr Beer: How does the design of a system that produces data that may be used for the purposes of legal proceedings, including criminal legal proceedings, affect the design of the system?
Charles Cipione: I’m going to go back to the concepts of: the results have to be complete and accurate, and that’s kind of lifted from the accounting world, completeness and accuracy. Accuracy you can think of as at the very atomic level of everything is correct, everything is correct. Completeness means it’s correct for everything.
So you want to – so, for instance, if you have a set of transactions and you are trying to display perhaps a sub-set of them and do a manipulation, do some sort of calculation and then summary information on that, what you want to be able to prove is, number 1, I’ve started with the complete set of data. The way you do that is you show, perhaps not all of the data is used in the final analysis, but you show that there’s basically a waterfall of: here’s the data – here’s everything, here’s this set of data that is being used in the analysis, and here’s the set of data that is not being used in the analysis.
A completeness check on that would be the set that’s not used in the analysis adds up to £100, the set that is being used in the analysis is £110, and I can go back and confirm that, showing that the complete set is 110 pounds’ worth of data. That would be just a very simple view of completeness.
Accuracy would be: perhaps I’m calculating an interest rate on some of the transactions and I want to show that, you know, there’s basically there’s a manipulation of that. So perhaps I’m doing a time series, and over time interest accrues to a particular amount. I want to be able to show that that calculation is accurate. So I want to show that it’s accurate at the detail level, and I want to show that it’s complete at the very highest level, so that there’s no doubt that the results of the reports that I’m doing are complete and accurate.
I can say that they are complete and accurate, and you can either believe me or not believe me. But, if I have a system or a process that is both transparent and auditable, it allows anyone that wants to look at it to confirm for themselves that, okay, I’m looking at the process that happened and, for purposes of transparency, that might be looking at the code that I used to do all the manipulations.
Alternatively, if it’s like a multi-step process and you want to see the waterfall going from step 1 to step 2 to step 3 and what the results are for the complete set of data, having all of that recorded from step 1 to step 2 to step 3 to step 4 and available for review, that would be an auditability check.
So if I have one of them, it probably would provide comfort to everyone. If I have both of them, it just increases the comfort, not only for them but for me because I want to be able to make sure, as I’m doing the process, that I’m doing it correctly. I want to make sure that there is a great amount of integrity in the results that I produce, especially if it’s in a commercial court or a business litigation court. I would expect that that level of custodianship is probably even more important in a criminal court.
Mr Beer: So does it come to this, that the answer to my question is that, yes, it does affect and it affects in the way that you have described the way that you design, build and run a system, if you know that the data from it may be used for the purposes of legal proceedings?
Charles Cipione: I would say it would affect it regardless of what system I’m using but, as the stakes go up, yes, it’s even more important to adhere to those principles.
Mr Beer: Thank you. Can we turn to the topics that you have addressed in your report, and they start on page 72. There is a series of topics – and we’re going to go through them one by one – that you have identified as being relevant and important.
Charles Cipione: Yes.
Mr Beer: The first of them is disclosed by the heading “Subpostmaster training experienced difficulties during National Roll-out.”
In paragraph 8.1.1, you address the importance of training. Can you help us at a general level, without addressing the Horizon IT System for the moment, how important is training when a new IT system is introduced?
Charles Cipione: Of course it’s very important for the users of the system to know how to use the system. The system is there for the users. The system is there to support the business processes that have been part of the requirements process for building the system.
The system could be great but, if the users don’t know how to use it, it loses all its value. So it’s critically important that the users be very comfortable with operating the system.
Mr Beer: In the case of the Horizon System, was there anything in particular that heightened the need for such training?
Charles Cipione: I would say generally my statement is a blanket statement. People need to – you know, users need to know how to use the system. I believe that there might be certain circumstances in the user community for the Horizon System that probably heightened the attention that might need to be spent on making sure that this user community is attended to properly.
The first one is my understanding is that, prior to the Horizon system, this was a manual process. So not only are you doing a change of process, you are also introducing the concept of technology to assist in the process. So any time you change that could be a challenge for some people.
But going from a very paper-based process to a computer-based process just adds another dimension of unknown to the formula, and it must be accounted for.
Mr Beer: This was happening at the close of the last century.
Charles Cipione: Yes, in the last millennium.
Mr Beer: Yes. You highlight in the last sentence of paragraph 8.1.1:
“It is apparent from reading the ICL monthly reports that there were significant problems in training the SPMs as they adopted the Horizon IT System.”
On the basis of what you read, looking at it generally, is it fair to say that was consistent theme throughout period that we’re looking at?
Charles Cipione: Yes.
Mr Beer: That was recorded in the ICL reports at a significant level of concern?
Charles Cipione: Yes.
Mr Beer: In paragraph 8.1.2, you make the point that this was not just a challenge of training users on a new system, it was actually a challenge of training users on computers in general. Is that a product of the age that we’re talking about?
Charles Cipione: Absolutely.
Mr Beer: You extract a quotation from, I think, material published by Fujitsu as part of its promotional and sales literature and, in the last five lines of the quotation from their promotional material, you say, or they say:
“Training was provided to 63,000 staff members from the ages of 16 to 87 with various skills … Approximately 5,000 calls were received each week by the Helpdesk, due to the counter staffs’ lack of computer experience.”
Is it on that basis that you make the point that the group of people that were being trained naturally had a variable level of skills?
Charles Cipione: Yes.
Mr Beer: 5,000 calls a week with a cohort of 63,000, did you draw any views on that?
Charles Cipione: That’s a high rate, for sure. I think it’s a function of, like I said, not only are we changing the system from paper-based to computer, but we do have a wide spectrum of what I would expect to be some people are very computer savvy at that point, but there might be some people that have moderate abilities in computers, might be some people that don’t have any ability to use computers, and there might be some people that are very resistant or just don’t want to use computers.
That’s a wide range of users to try to accommodate, and I would expect that that probably was part of the calculus that went into those numbers.
Mr Beer: You tell us in paragraph 8.1.3 that Pathway was aware of the importance of training the subpostmasters and that that was recognised by them early on, and you note that Pathway itself noted in April 1999 that the subpostmasters were facing difficulties in transitioning from a paper-based balancing process to an automated balancing process, and that ICL Pathway responded by saying that a greater emphasis should be placed, in training, on the practical experience of balancing. That was the answer that was given at the time; is that right?
Charles Cipione: Yes.
Mr Beer: Did it appear that that was successful, i.e. the answer from ICL saying a greater emphasis must be placed on practical experience of balancing, given the reports of difficulties in balancing persisted throughout 1999?
Charles Cipione: I apologise, Mr Beer, what was the actual question?
Mr Beer: Did it appear to you that the response to the problems that subpostmasters reported of facing difficulties in balancing, namely, to say, “Well, a greater emphasis must be placed on training, on practical experience of balancing”, was successful, given the reports of difficulties in balancing persisted throughout 1999?
Charles Cipione: I would say that it didn’t appear to alleviate the problems.
Mr Beer: Can we look at some examples over the page, please, in the extracts from the management reports, including ICL Pathway’s monthly report. In fact, just before we do so, could I look at what you said in paragraph 8.1.4 at the foot of the previous page.
You tell us in the last few sentences that what we’re about to look at are verbatim extracts that you have highlighted some matters in bold, that the views expressed are those of the authors, being principally ICL Pathway documents, but in some cases joint ICL and Post Office documents.
Charles Cipione: That is correct.
Mr Beer: If we then go over the page, please, do you in the second column of this table make clear whether this is an ICL document that you are quoting verbatim from, or a jointly issued document from ICL Pathway and the Post Office?
Charles Cipione: Yes, it’s clear. So, for instance, the first line, ICL Pathway monthly report April 1999, that’s an ICL Pathway-authored – a solely ICL Pathway-authored document.
Mr Beer: The rest of that page, I think, is all the same?
Charles Cipione: That’s correct.
Mr Beer: Then, if we just skip over the page –
Charles Cipione: Yes.
Mr Beer: – and look at the first entry.
Charles Cipione: Yes, the first entry there, ICL and Post Office monthly joint implementation report. So that would have been co-authored by ICL and the Post Office.
Mr Beer: Then I think the rest of the documents on the page are ICL Pathway alone-issued documents?
Charles Cipione: That’s correct.
Mr Beer: I think we can see that in evidence, not only by your description of the report as being an ICL Pathway monthly report, but also from the extracted text. If you look at the second report for May, Pathway wrote:
“POCL are shaping up to hit us on service level agreements and training.”
Charles Cipione: Yes, that would have been ICL Pathway’s view.
Mr Beer: So the “us” there is the ICL Pathway view?
Charles Cipione: Yes.
Mr Beer: Was there anything in the content of the reports that suggested, where they were authored by ICL Pathway themselves, these monthly reports, that POCL were provided with copies of them?
Charles Cipione: I don’t believe I ever saw any indication of that.
Mr Beer: From the other evidence that you read from PinICLs, PEAKs, KELs or other documentation that you read, was there anything in that which suggested that those ICL Pathway monthly reports were provided to POCL?
Charles Cipione: No.
Mr Beer: Now that formulation that we read at the end of paragraph 8.1.4, if we just go back to that, at the foot of the page, is the same formulation, give or take a few words, that you use in succeeding sections of your report, when you’re about to introduce the table of verbatim extracts from PinICLs, PEAKs, KELs, monthly reports?
Charles Cipione: Yes.
Mr Beer: And does what you just said apply equally to all those succeeding sections?
Charles Cipione: It does.
Mr Beer: Thank you. Go back to the substance then, if we can just briefly look over the page, please, at September ‘99. Scroll down, please. The first entry ICL records:
“Although national roll-out rates have risen to 2,000 (sic) Post Offices per week, the level of issues occurring on installation day and the level of training scheduling failures puts achievement of the 300 offices per week roll-out rate required in 2000 at risk. Knowledgepool are introducing new scheduling software and a plan of activity to remove/reduce the causes of the other issues is being put in place for the November to January break in National Roll-out.”
Can you help us who were Knowledgepool?
Charles Cipione: I believe Knowledgepool was the partner with ICL that assisted with the training of the SPMs.
Mr Beer: Essentially a subcontractor of ICL Pathway that provided the training?
Charles Cipione: Yes.
Mr Beer: We can see that at the second entry for September 1999:
“ … currently a serious issue relating to the scheduling of training events within the implementation programme. The training scheduling system of Pathway’s training subcontractor, Knowledgepool, has been struggling to scope during the early part of national roll-out, although a planned system replacement was imminent.”
So this isn’t Horizon itself. This is a separate system used for scheduling training; that’s how you understood it?
Charles Cipione: Yes.
Mr Beer: During September the training scheduling system crashed resulting in a loss of data and some data corruption, and then we can read the rest.
Then, if we go over the page, please, the joint report covering a period of September ‘99 to nearly the end of October ‘99, training continues to cause major difficulties, and then in May, the two lots of May 2000 entries, if you just cast your eye over those, please. The first:
“POCL perception of SLAs and training, and also of our commercial attitude to risk-taking on new business: all negative as epitomised by the recent Dave Miller letter. Risk remains that POCL will extract commercial concessions out of us …”
Did you track that through, the suggestion that POCL would seek money from ICL Pathway in relation to the provision of training?
Charles Cipione: When you say, did I track that through –
Mr Beer: Yes, did you see that in later documents?
Charles Cipione: I believe that that was a running commentary, yes.
Mr Beer: We can see in the entry then of December 2000:
“A settlement for the projected shortfall in training courses against the contracted, number arising from low course-occupancy levels, has been agreed with the Post Office. As part of a package to achieve relaxations against existing service level agreements, pathway will pay the first £1 million of the training shortfall.”
That’s what I was –
Charles Cipione: Yes.
Mr Beer: – referring to.
Did what you read in these reports, these management reports, reflect what you found in the PinICLs when you came to read them relating to training issues?
Charles Cipione: Yes.
Mr Beer: If we look at the foot of the page at 8.1.5, you start a summary of some of the PinICLs and include verbatim extracts from them. I’m not going to go through them, but you summarise them in 8.1.5:
“A review of the PinICLs and PEAKs [I think these are just PinICLs in fact] reinforces the theme that the subpostmasters were reporting that the lack of training was problematic in the execution of business activities. Additionally SSC staff [remembering back, that’s the third tier of support] were also raising concerns about the ineffectual nature of training.”
You embolden some sections. You then set out in paragraph 8.2 a series of PinICLs. As I say, I’m not going to go through them. Can we go forward to page 76, please, and look at 8.1.6, which is towards the bottom of the page. You say that you:
“… surveyed the PinICL and PEAK population for any final defect cause being assigned ‘General - User’ or ‘General - User Knowledge’ …”
Can you just explain what you mean by surveying the population for any final defect cause?
Charles Cipione: Certainly. So at the beginning of our conversation today, we talked about the analytics work stream and evaluating the PEAKs and PinICLs information, and a part of that analytics work stream was to help identify certain data elements that were contained within the PEAKs and the PinICLs.
One of those data elements was a defect cause and a defect cause is essentially the – as an error was raised and entered into the PEAK or PinICL system, it was evaluated, and part of that evaluation was ICL would make a determination on what the cause of the particular defect that raised that error was.
This is a representation of that analysis. So, looking through a population of PEAKs and PinICLs that we had in our possession, we looked for the final defect cause. So perhaps it might be good for me to talk about that.
So the defect cause has the opportunity to change values as the error is being investigated. It might be thought that there might be a particular, an initial evaluation that a defect cause is one thing, but upon further investigation it has the opportunity to change.
The analysis here is looking at what the final value of the defect cause was across the PEAK and PinICL population, and I identified two values that I thought were pertinent for this section.
Mr Beer: They are General-user and General-user knowledge?
Charles Cipione: Exactly.
Mr Beer: That resulted, in the period you were looking at, 435 PinICLs being identified where those two values had been applied?
Charles Cipione: That is correct.
Mr Beer: You say:
“Please keep in mind that the SMC was supposed to resolve user issues. These PinICLs and PEAKs were promoted to the SSC.”
What did you mean – what sits behind those two sentences?
Charles Cipione: So we’ll talk about in greater detail, I’m sure, as the conversation continues. But the SSC who is in charge of the PEAKs and PinICLs, that’s more of a technical error evaluation and remediation system. User questions should have been identified by the Helpdesk or by the SMC, which are the first and second levels of basically help to the users. Only real technical errors are supposed to arrive for the SSC to evaluate but, if it does get to the SSC, they raise a PinICL and it starts to be documented.
So what I’m trying to convey here is that this is what actually didn’t get resolved at the first and second level. This population could be much greater if we went and looked back at the actual Helpdesk tickets, because the Helpdesk is what precedes anything that might get into the PEAK or PinICL system.
Mr Beer: The 5,000 calls a week?
Charles Cipione: Yes, exactly.
Mr Beer: What’s the significance of this finding in 8.1.6?
Charles Cipione: There are a lot of problems with user knowledge on how to use the system.
Mr Beer: In 8.1.7, you say:
“This figure indicates a wave of user issues in September ‘99, March 2000, June 2000 and November 2000 during the national roll-out period.”
If we go over the page, please, if we can blow that up slightly, you depict the overall volumes between, I think, May ‘99 and December 2000.
Charles Cipione: Yes.
Mr Beer: The spikes that you speak about, September ‘99, which we can see before, October ‘99, March 2000, June 2000 and November 2000 – I suppose, July as well – the colour coding on the right-hand side, does that refer to the final defect cause?
Charles Cipione: Yes.
Mr Beer: So the thing that you’ve just spoken about?
Charles Cipione: Exactly.
Mr Beer: Given the process of progress in national roll-out, did you think about whether this is a function, these PEAKs, of the increasing number of users being admitted to the system?
Charles Cipione: It absolutely is part of the function, yes.
Mr Beer: Did you draw any overall conclusions from what you saw in the monthly reports and the PinICLs, as to the training difficulties that were experienced during national roll-out?
Charles Cipione: Just the general concept that the training didn’t seem to go swimmingly well. The user base had a lot of problems with understanding how to use the system or, throughout the national roll-out period, that’s the conclusion that I would draw.
Mr Beer: Thank you. Can we turn to section 9, please, of your report on page 78. The heading “Hardware issues were problematic during national roll-out”, and if we read 9.1.2:
“In the national roll-out of the Horizon IT System, there was a discrete period (August 1999) where hardware issues rose to the level of being ‘ a serious Acceptance Incident’. According to ICL Pathway’s monthly reports some issues persisted through October 1999, but appear to have subsided to an acceptable level by January 2000.”
Now, the material that you relied on to say that, I think, all came from the ICL Pathway monthly reports which you set out in table 9.1; is that right?
Charles Cipione: Yes.
Mr Beer: If we go forwards, please, to page 80 and read that very short paragraph at the top of the page:
“It should be noted that in May 2000, there were still hardware issues being raised in PinICLs.”
Then, if you then look at the table at 9.2, you’re essentially checking what you read in the monthly reports again against the PinICLs; is that right?
Charles Cipione: That is correct.
Mr Beer: Now, despite what was said in the ICL Pathway monthly report about the issue subsiding to an acceptable level at the start of 2000, you record that in May 2000 there were still hardware issues being raised in PinICLs; is that right?
Charles Cipione: That is correct.
Mr Beer: Did you conduct a survey of PinICLs and PEAKs between July 1996 and the end of 2000, where the product at fault in the PinICL or PEAK was shown as desktop or hardware?
Charles Cipione: Yes.
Mr Beer: If we turn over the page to page 81, please, and just scroll down to 9.1.5 and the table underneath, did you, using that product-at-fault value, find that there were 1,281 PinICLs or PEAKs that had that value ascribed to them as the final fault?
Charles Cipione: Yes.
Mr Beer: Did you set those out in your table between the period I mentioned, July ‘96 and end 2000, at 9.1?
Charles Cipione: Yes.
Mr Beer: Does that show a spike during the national roll-out period?
Charles Cipione: Yes, it does.
Mr Beer: Again, that might be a function of the fact that there were more users being brought online?
Charles Cipione: Absolutely.
Mr Beer: If we look at 9.1.6, you say you again surveyed the PinICLs AND PEAKs for product groups listed in the next figure’s legend, where the product-at-fault value was hardware, ISDN, ISDN adaptor or driver.
Just explain again what you are doing there.
Charles Cipione: So there’s a value within the information that I received called “product at fault” and, to the extent that it existed and we identified a value for that, what we did was we basically pulled out any product-at-fault values that were either hardware, ISDN or ISDN adaptor/driver, all of which I would associate with a hardware issue.
Mr Beer: Do we see, if we go over the page, in 9.2, a similar pattern of a PEAK during national roll-out –
Charles Cipione: Yes.
Mr Beer: – in table 9.2?
Charles Cipione: Sorry, yes.
Mr Beer: In 9.1.7, you tell us that you looked at 332 KELs of which about 10 per cent were coded as responsive, i.e. the nature of the KEL was concerned with a hardware-related issue; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: Looking at all of those sources, the monthly reports, the PinICLs and PEAKs, and the KELs, did you form any view overall, as to the preponderance of hardware issues during national roll-out?
Charles Cipione: The hardware issue – first of all, there absolutely was an issue with hardware during national roll-out, and it impeded the roll-out certainly.
Mr Beer: Could such issues affect the reliability of data held on the system, and could such reliability issues lead to what would seem to be inexplicable financial imbalances?
Charles Cipione: That would – yes, there’s a possibility that that could happen, absolutely.
Mr Beer: Is it normal that there are hardware issues when you roll out a big system?
Charles Cipione: Yes, and we need to remember the time range that we’re talking about. I would say, over the last 25 years, the reliability of hardware used in technical systems has improved drastically. It wasn’t as good. It certainly wasn’t – the hardware wasn’t as good in the late ’90s as it is now, as far as reliability and durability is concerned. So I am extremely aware of that.
Notwithstanding that statement, that probably was little comfort to anyone that was experiencing the hardware issues during that time, and it definitely posed a problem for the national roll-out.
Mr Beer: Could anything have been done to warn or inform end users, subpostmasters, of the issues that might be caused by hardware problems; so hardware-generated keystrokes, for example, or phantom transactions?
Charles Cipione: Yes. So to the extent that ICL Pathway was aware of such a specific thing as that, definitely communications could have been made. I suspect that that was a small fraction of what I’m showing here. I think what I’m showing here is more of a catastrophic, holistic failure of either the communication, the ISDN line, or the counter set-up. It could be that both your hard drives failed. You know, unfortunately that’s just things that happened during that time. I mean, drives could fail right now certainly also, but the prevalence of that was much greater 25 years ago.
Mr Beer: Overall I think you told us that you formed the view that hardware issues caused problems, and problems for subpostmasters, and it was problematic for them during the national roll-out.
Charles Cipione: Absolutely.
Mr Beer: Can we turn to section 10, please, page 83, where you address the disconnection of many Post Office branches from the central system during national roll-out. You tell us that in 10.1.1 that:
“The ambition of Horizon was to allow branches to communicate their information to a central system” –
Reminding us of an earlier passage in your report, the Horizon campuses.
“It also allowed for software and reference data updates to be distributed from the campuses to the branches.
“To accomplish this … a telecommunications system was incorporated into Horizon.”
This depended on ISDN lines or satellite links being installed at each branch with BT or Energis providing the backbone infrastructure to utilise this hardware. It relied also on each branches equipment to be available for polling.
Can you tell us what polling means, please.
Charles Cipione: Certainly it’s basically having a remote computer being available to communicate with a central computer, and the polling is usually described as when the central computer reaches out to the remote computer to see if it’s available to communicate. So that would be polling.
Mr Beer: How does that work? Can you explain in a little more detail how that works.
Charles Cipione: Certainly. So in this system the network was not constant – always on. It was not like systems are now when you’re – well, even when you’re on your browser, you’re not really actually always on necessarily.
But effectively what they were – the mechanism that they were using was they had a communications link, either through a satellite or through an ISDN line, that utilised BT or Energis’ backbone or the other service providers, which allowed the movement of data between the counter information at the branch and the data servers at Bootle or Wigan.
What would happen is every night the counters needed to basically communicate with the central branch to accumulate all the transactions that happened, right? So that would be the direction of: I’ve conducted my business. Now I want to make sure that all of the records of those transactions are managed centrally at the data centres.
In order to accomplish that, the data centre and the counter needs to communicate. Since it’s not a persistent connected communications link, what would happen is a polling would have to occur. The central system would need to reach out to each one of the counters and say, “Number 1, are you even there and, if you are there, I need to grab some information from you. I need basically to collect all the transactions that you performed that day so that I can do the downstream processing of all that information.” So effectively that’s what the polling information is.
There are occasions also where the central system needs to push information down to the counters. That might be: I’m updating reference data which is – if you remember, it’s part of the system that allows the system to work correctly and, in order to maintain synchronisation of how each system works, all of that information needs to be passed down to the counters, as well as just straight software updates, that would need to be passed down to the counters.
Mr Beer: You tell us in 10.1.3 that:
“The Monthly Reports indicate that throughout ‘98 and ‘99 Pathway was concerned with their ability to effectuate this design feature: they were concerned with BT’s coverage of the UK as well as other technical issues related to their standards.”
Is that concern by ICL Pathway before national roll-out?
Charles Cipione: Yes.
Mr Beer: This is even before national roll-out. Then if we look at 10.1.4:
“During the national roll-out these problems were realised.”
So, true enough, ICL Pathway’s worries turned out to be correct?
Charles Cipione: Yes.
Mr Beer: “Hardware, network availability, and user issues combined to create a situation where ICL Pathway was occupied with a higher than expected amount of non-polling branches.”
Given what you have said, that there was concern before national roll-out about ICL Pathway’s ability to effectuate this design feature, i.e. to ensure polling with hardware installed at the branch, what do you mean by: it was higher than expected amount of non-polling branches?
Charles Cipione: I would imagine, in any contingency planning that anyone was doing with a system that was designed like this, that they would expect that some counters wouldn’t be available for some periods of time. But based off of the reading that I performed, it was much higher than they expected as far as non-polled counters.
Mr Beer: You say in the third sentence:
“This was problematic because Horizon relied on this … design aspect to not only collate and centralise information … but … also for efficient updates of software to the branches.”
What level of problem is this?
Charles Cipione: It’s a giant level of problem. Let’s take it to an extreme. Let’s say that I never am able to poll a counter. Effectively that counter is out of the network. So, if I am not able to communicate between the counter and the data centres, that’s breaking the system. That’s taking all of functionality that was purported to be in the Horizon System and throwing it out on a counter-by-counter basis.
Mr Beer: For bug fixes, did that rely on the ability to poll?
Charles Cipione: Yes, yes. Any communication between the central servers and the counters relied on that communication mechanism, which I’m referring to as the polling. The polling is actually: are you there? Once you get an affirmative answer back, then you can move information between either the counter and the data centre or the data centre and the counter.
Mr Beer: You say in 10.1.5:
“Additionally ICL Pathway was compelled to raise and resolve an issue for any branch whose non-polled status was 24 hours in duration.”
Charles Cipione: Yes.
Mr Beer: Just explain what that means.
Charles Cipione: So I believe this is related to an SLA requirement for ICL Pathway, just as part of the requirements process. They needed to make sure that they were polling different counters, and that’s regardless of why – you know, there could be things that were completely out of ICL Pathway’s control that would stop that polling from happening, such as the counter wasn’t turned on. If that counter’s computer is never on, it’s never going to be polled and, unless ICL Pathway sends someone out to turn it on, it’s never going to be turned on. So there were definitely some aspects of that that were outside of their immediate control.
Mr Beer: You set out extracts from the monthly reports that essentially establish what you said in your last few answers. Can we go on to page 86, please, and look at the table 10.2. Could explain what this table shows, please.
Charles Cipione: Certainly. So this table is describing all of the days in the month of November of 2000, and it is creating a set of categories of how many counters or how many offices weren’t able to be polled across a different set of days. So, for instance, we see the first figure of 86. On November 1, 2000, there were 86 counters that had not been polled in a day.
Mr Beer: You use language interchangeably of counters and branches, and counters and offices. Is it counters or is it offices?
Charles Cipione: This says offices. So maybe we should also refresh our memory on exactly how the Horizon System worked at that time. So there might have been many particular counters at each office, but there was only one counter that was hooked up to the communications system that allowed for polling. So this would be office, not necessarily counter. This would be per branch or per office.
Mr Beer: You were telling us that, for 1 November, there were 86 offices that did not poll for one day?
Charles Cipione: Exactly.
Mr Beer: There were 83 that hadn’t polled for 2 or 3 days?
Charles Cipione: Yes.
Mr Beer: There were 28 that hadn’t polled for between 4 and 9 days?
Charles Cipione: Yes.
Mr Beer: So it goes on?
Charles Cipione: Exactly.
Mr Beer: Did you draw a conclusion from this level or extent of non-polling offices and the duration?
Charles Cipione: This was an issue. Especially as we look further into November, you see the numbers becoming very high there. So, even if there were, you know, 20,000 offices – sorry, 20,000 branches – let’s pretend that there’s 20,000 branches live in the network – that’s a significant – you know, towards the end of November that’s a significant portion of those branches that are not communicating with the central data servers.
Mr Beer: Does this show a consistently serious problem across the entire month essentially?
Charles Cipione: Yes.
Mr Beer: You then again cross-refer to what the monthly reports had shown to the PinICLs and PEAKs; is that right?
Charles Cipione: Yes.
Mr Beer: If we look at the foot of 87, page 87, at 10.1.8, you say:
“I surveyed the PinICL and PEAK population for entries where the Product at Fault contained ‘ISDN’ or ‘VPN’ within their values.”
That’s because they are both related to connectivity.
“This query resulted in the 4,733 entries shown in the figure below, with their specific Product-at-Fault value shown in the legend. This problem manifested during the national roll-out period.”
Then over the page, please, to the graph. The colours on the right-hand side or the key on the right-hand side, I think, shows that the preponderance of the problems related to the shade of blue that relates to ISDN; is that right?
Charles Cipione: Yes.
Mr Beer: The fourth colour down on the right-hand side?
Charles Cipione: Yes.
Mr Beer: Again, 4,733 product-at-fault values with ISDN or VPN, did you draw any conclusion as to whether that signified anything about the size or severity of the problem?
Charles Cipione: It indicates that there’s absolutely an issue that was related to the physical connection or the implementation of the Horizon package at this time.
Mr Beer: Thank you.
I’m going to skip over for the moment sections 11 and 12 of your report and go straight to section 13, please, on page 98 where your heading of the issue is:
“The persistence of reference data management degraded the integrity of Horizon.”
Can you explain, before we get into the detail of this, what your overall finding was.
Charles Cipione: My overall finding was – so just to remind everyone, the reference data is a component of the system that I referred to at my first hearing as data-driven logic. It could represent something as simple as just a price list, you know, a first-class stamp costs this much. That information was not hard coded into the Horizon System. It was passed to the Horizon System via what I’m referring to as reference data.
Now, my understanding is there are a lot more complicated partitions of the reference data that I won’t go into now, but it was essential for the Horizon System to operate correctly for that to have integrity and to be consistently completely and accurately delivered to each one of the branches in order for the Horizon System to operate.
What I’m saying here is there were a lot of issues that I read about that were referring to the integrity of the reference data as it pertained to the actual operation of the Horizon System.
Mr Beer: I think on the last occasion you identified, or you told us that a design of a system that utilised data-driven logic was in principle a good thing.
Charles Cipione: Absolutely.
Mr Beer: You say in 13.1.3:
“The advantages of data-driven logic rely upon its custodianship. If the ‘data’ in the data-driven logic is not timely, accurate or complete, the system it supports will not operate as intended.”
I think that speaks for itself.
Charles Cipione: Yes.
Mr Beer: You then in 1.4 and 1.5 set out issues raised by ICL Pathway as early as early 1997, concerning POCL’s reference data incorporation into the system. You say:
“By late ‘97 ICL Pathway characterised its contractual obligations regarding reference data as poorly defined but acknowledged the significance of the issue as crucial.”
What do you mean by that?
Charles Cipione: So, if this is a vital component of your system, you have to make sure that it’s right. What I’ve read in the materials that were provided to me was that this was a split responsibility between ICL Pathway and the Post Office Counters Limited.
Ultimately, a lot of the information needed to be sourced from POCL. I don’t think all of it, but a lot of it was sourced from POCL. What I’m trying to say here is that I don’t know that the communication of requirements or the – I believe that what ICL Pathway is saying here is they don’t think that the Post Office is understanding the importance of this issue as ICL Pathway is trying to develop and deliver a system.
Mr Beer: Thank you. You say in 1.6 that you are going to extract from the monthly reports the data that supports what you have just said. Then, if we go over the page, please, to the foot of the page – keep going, please – sorry, two more pages to 102, and then foot of the page, please.
In 13.1.7 you say:
“A review of the PinICLs and PEAKs supports the contention that Reference Data was a cause of problems in the Horizon IT System.”
You set those out in your table at 13.2. Go over the page, please, and then over the page again to page 105. You say that you:
“… surveyed the PinICL and PEAK population for any Product-at-Fault value where Reference Data was indicated.”
You found 1,863 such PinICLs or PEAKs, and that led you to the conclusion that reference-data problems began manifesting themselves in 1998 and were prevalent during the national roll-out period.
Can you just identify for us in the table – thank you – it’s the dark blue, second from bottom; is that right?
Charles Cipione: Yes. So the dark blue second from bottom says “reference data” explicitly, yes.
Mr Beer: But the other product-at-fault values identified on that table are product-at-fault values that also in your view relate to reference data?
Charles Cipione: Yes.
Mr Beer: I think they show a persistence of the issue right until the end of 2000; is that right?
Charles Cipione: That is correct.
Mr Beer: How serious or problematic is this?
Charles Cipione: This is a big problem. This is a main component of the system. If the reference –
Mr Beer: I missed that.
Charles Cipione: This is a big problem. This is a serious problem. If this is not done correctly, the system is not working correctly.
Mr Beer: Going back to 13.1.8, you say:
“Interestingly, [I suppose that’s a relative value] a Product at Fault value of ‘POCL Reference Data’ seems to appear in February 2000 and from that point forward occupies a significant portion of the chart.”
Without teasing you too much, why did you find it interesting?
Charles Cipione: I found it interesting that, I believe, ICL Pathway was trying to clearly delineate whether they thought this reference-data issue was a Post Office issue or not a Post Office issue.
Mr Beer: I see. So they are being more precise as to the attribution of where they believed the reference-data issue arises from and attributing that to POCL?
Charles Cipione: That is the conclusion I’ve drawn, yes.
Mr Beer: Is that the sky-blue colour in the chart at 13.1 –
Charles Cipione: Yes.
Mr Beer: That’s why we don’t see it before because it’s a new product-at-fault value?
Charles Cipione: That’s right. Hypothetically, before that it was still could have been caused by POCL, but they weren’t tracking it at that level of precision, as you say.
Mr Beer: That brings us to the end of section 13 of your report. Sir, might that be an appropriate time to take the morning break?
Sir Wyn Williams: Yes.
Mr Beer: We lost you at the end of your sentence there.
Sir Wyn Williams: I muted myself too quickly. I said that’s fine, Mr Beer.
Mr Beer: Thank you very much, sir. Can we say half past, please.
Sir Wyn Williams: Yes, of course.
Mr Beer: Thank you.
(11.17 am)
Mr Beer: (A short break).
(11.31 am)
Mr Beer: Good morning, sir. Can you see and hear us?
Sir Wyn Williams: Yes, thank you.
Mr Beer: Mr Cipione, can I go back before we move on to section 14 to something you said in section 13 of your report. If we just go back to 13.1.1 on page 98, you say in 13.1.1:
“This design feature [that’s data-driven logic] it’s benefit was efficiently to update the Horizon IT System’s functionality without the need to develop, design, test, and deploy new versions of the software.”
I think you are saying that as a positive thing there.
Charles Cipione: Absolutely, yes.
Mr Beer: Would you accept that the use of reference data in this way, whilst it lowered the need for development, design, testing and deployment of new versions of the software, in practice it may have led to a need for increased testing?
Charles Cipione: If the integrity of the reference data or the mechanism for distributing the reference data wasn’t good, it could absolutely create a situation where there are a lot of errors in the system, and it might not be readily apparent where those errors are arising from. So that, in effect, could cause more testing, more cycling, more uncertainty around the causes and remediation efforts required to make the system work properly.
Does that answer your question?
Mr Beer: Yes, I think it does.
Can we turn to section 14 of your report, please, page 106 and your subject heading is:
“The Horizon IT System Helpdesk was often the root of Service Level Agreement issues with POCL.”
Again can you just explain in overview what this section of your report is concerned with.
Charles Cipione: This section is concerned with the front-line support that was provided to the subpostmasters and subpostmistresses at the branches that were using the Horizon System. There was – one of the service level agreements, probably many of the metrics in the service level agreements, required ICL Pathway to have a Helpdesk that was responsive when calls came in, and that’s what I’m referring to here.
Mr Beer: Here you’re referring to the HSH Horizon System Helpdesk. That’s the first line of support?
Charles Cipione: That is the first line of support.
Mr Beer: You say that in the third line of 14.1.1:
“The monthly reports discussed the failings of which HSH, in regard to SLA [failings] for a significant amount of the review period. Concerns were first raised in September ‘98 and carried through the national roll-out.”
On your review of the material, did those concerns persist at the same level throughout that period?
Charles Cipione: They were a consistent topic of discussion in the material I read throughout the period.
Mr Beer: You say:
“The same issues that triggered SLA concerns also ‘dented’ confidence from POCL.”
Where did you get that information from?
Charles Cipione: The monthly reports.
Mr Beer: So is that ICL saying that those issues dented POCL’s confidence, so their perception of POCL’s confidence in them?
Charles Cipione: Yes, those are ICL’s words.
Mr Beer: “In May 2000, ICL declared an ‘own goal’ …” I think that’s the word in the report itself, or words in the report itself; is that right?
Charles Cipione: That is correct.
Mr Beer: “… based on Horizon System Helpdesk performance.” They replaced their management team, and improvements were noted in June 2000.
Charles Cipione: Yes.
Mr Beer: You then set out in your table at 14.1, on that page and the next, extracts from the monthly reports that substantiate the conclusions that you had reached. Then at the foot of the next page, you tell us that you didn’t examine the PinICLs and PEAKs to substantiate or undermine what was said in the monthly reports. Just explain why.
Charles Cipione: I wouldn’t expect any discussion of front-line SLA response rates to exist within the PinICL or PEAK documents.
Mr Beer: Can we turn to section 15 of your report, please, page 108, where your conclusion is essentially in the section heading:
“The Horizon SMC was frequently cited for not properly filtering calls to the SSC. This lack of filtering delayed the SSC from resolving technical problems.”
So you are here addressing the second to the third level support lines; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: You remind us in 15.1.1 of the error-escalation process where users were directed to level 1, the HSH first.
“The SMC [at level 2] was responsible for determining if the problem required the SSC to become involved. If the issue was deemed necessary for escalation … it would then be recorded in the PinICL system.
“The SSC was responsible for the maintenance of PinICLs”.
What’s the importance of a tiered escalation process such as this?
Charles Cipione: It would create a very efficient process to handle any issue that was raised by anyone using the Horizon System. The efficiency I’m referring to here relates to: a user is having a problem. That problem could be that the user doesn’t know how to use the system or something very simple, in which case it would be good to have at the first period of triage someone there that could help with very simple answers to the user community. That would have been the HSH, which was the first line.
But it’s not efficient as far as the Helpdesk management process, or the HSH, to have a global understanding of every issue that might or might not come up. So they need to have someone to refer to if their – because usually the first line of Helpdesk is usually provided with a number of scripts that allow them to diagnose information that was received by the callers, and either help the caller immediately, or at some point that end of the script it’s, “Okay, I don’t know enough about the system right now. I need to pass you on to the next level.”
The SMC is the next level who, theoretically, are a little bit more knowledgeable about the technical issues of the system, as well as the just general usability of the system. It’s their responsibility to make sure that calls that get beyond the initial Helpdesk are created or routed correctly.
So, if you recall, in my first testimony, the SSC is not really in charge of any hardware issues. So, if a call gets to the SMC, and it’s clear to the SMC that this is hardware issue, I believe that there was a different group that helped the hardware issue. So perhaps the front line would not know exactly where to send that, but the SMC would.
The SMC was supposed to only send calls on to the SSC, if they were a technical – a non-hardware issue technically related to the Horizon System. So that was their purpose and that is generally a good construction of a helpdesk process.
Mr Beer: What did you see in the monthly reports when the topic of the SMC filtering calls was discussed?
Charles Cipione: When I did read about the SMC in the monthly reports, it was usually in regard to the SMC was sending calls to the SSC that they should have been able to handle, or the Helpdesk should have been able to handle, or certainly was not the responsibility of the third line of support, the SSC, the technical portion of the Helpdesk.
Mr Beer: What are the consequences of that?
Charles Cipione: The consequences of that are: now you have a group of people at the SSC – that’s the third line – who are responsible for resolving actual technical issues of the Horizon System. That’s a difficult job for multiple reasons.
The situation that is created when the SMC is not filtering calls correctly is now they are having to deal with not only technical issue calls but non-technical issue calls, but they don’t know that necessarily when they’re getting it. If the SMC promotes an issue to the SSC, they think it’s a technical issue. So they start investigating it and probably spend a good amount of time on it. If they find out that this was something that should have been handled by the first or second line of support, their attention has been diverted from actual technical issues to non-technical issues that should have been resolved prior to it getting to the SSC.
Mr Beer: I think you found, you tell us in 15.1.4, that that problem persisted throughout the national roll-out?
Charles Cipione: Yes.
Mr Beer: You also, if we go to page 110, please, examined at the bottom half of the page extracts from PinICLs. The heading to that table says “Verbatim extracts from monthly reports”. I think that should be from PinICLs and PEAKs?
Charles Cipione: Yes.
Mr Beer: To see whether that which you had read in the monthly reports was supported, and I think you found that it was?
Charles Cipione: It is.
Mr Beer: We can see some of the things which I suppose the SSC were writing emboldened. The first entry, 27 May ‘99, “Why was the call sent to the SSC?” 17 June ‘99, “I’m unsure why this was sent to the SSC.” 13 October ‘99, “I don’t understand why this call has been sent to the SSC.” So it goes on right up until December 2000.
Charles Cipione: Yes.
Mr Beer: So despite I think the issue having been recognised it persisted; is that right?
Charles Cipione: That is correct.
Mr Beer: Can we turn, please, to section 16 of your report.
“The SSC was overwhelmed with PinICLs and PEAKs but was earnest in its effort to perform its duties.”
Is this related to section 15 that we’ve just looked at?
Charles Cipione: Yes. Yes, this is the third line. We discussed them a little bit in the prior section.
Mr Beer: In terms of the SSC being overwhelmed with PinICLs, was that related to the SMC failing properly to filter calls up?
Charles Cipione: That was an aggravating condition.
Mr Beer: What were the other causes of the SSC being overwhelmed with PinICLs and PEAKs?
Charles Cipione: There were a lot of errors being raised about the Horizon System.
Mr Beer: You record in 16.1.2:
“The ICL Monthly Reports often call attention to the workload of the SSC.”
You reviewed the PinICLs and PEAKs and recognised the complexity of some of the issues that SSC was responsible for resolving. What did you mean by that, that you recognised the complexity?
Charles Cipione: So just – you know, in the course of doing my review, of course I looked at quite a few of the PinICLs and PEAKs, and I needed to gain an understanding, as best I could, of the issues that were being raised, meaning that there was a PinICL or PEAK created, and the remediation process that was performed to attempt to remediate these errors as they arose.
In the course of that review, I had to basically dive as deeply as I could into understanding many of these PEAKs or PinICLs. Through the course of that reading, it became clear to me that some of these issues were difficult to identify what the cause was and, therefore, reading the narratives, I just wanted to comment that it was – it was a good effort on the people – on the PinICLs and PEAKs that I read, for the people that for the most part – now this is not a blanket statement, but this is a statement for the ones that I reviewed – for the most part of the review process that I observed through reading the narratives of the PinICLs and PEAKs, there appeared to be a very earnest effort to try to resolve the issues.
Mr Beer: What are the parameters of that view in terms of when or for what period?
Charles Cipione: Are you talking what was the period of documentation –
Mr Beer: No, the view that there was an earnest attempt to resolve problems by the SSC.
Charles Cipione: So my parameters were: was attention being given to it? Did it seem to have been evolving? You know, was extra information needed? Was it provided? What was done with that information and, you know, was there some sort of resolution at whatever level achieved in those PinICLs and PEAKs?
Mr Beer: It’s my fault. You just answered what values did you use. My question was: over what time period did that conclusion relate to?
Charles Cipione: Sorry, yes. So the information that I was looking at was from 1996 through 2000.
Mr Beer: So the entire period?
Charles Cipione: Yes.
Mr Beer: You extract for us in 16.1. and the table at 16.1 parts of the monthly reports that support those conclusions. Then, if we go over the page, please, to 16.1.5 on page 114, which is at the foot of the page, you introduce your figure 16.1 by telling us that the following figures line shows the open balance of PPs, i.e. PinICLs and PEAKs, by day. Can you just explain in broad terms what the table is then?
Charles Cipione: So an open – so just to go through the life of one PEAK or PinICL –
Mr Beer: Yes.
Charles Cipione: – it’s entered into the system. There’s an entry date into the system. It’s worked on for one to many days. At some point the PEAK or PinICL has closed. So what this chart is showing is, as PEAKs and PinICLs are entered and relieved from the system, there’s a particular open balance on any particular date. This chart is showing what the open balance was of the PEAKs and PinICLs on any particular day throughout the period in the chart.
Mr Beer: Can we look at the chart and you speak to it.
Charles Cipione: Certainly. So we can see that the chart starts in the latter part of 1996, and the information as far as new PinICLs and PEAKs existing for entry into this chart runs through, you can see right there, at the end of 2000.
Mr Beer: Yes.
Charles Cipione: So it might be a little difficult to see, but there is a little bit of a different colour at the beginning of 2001.
Mr Beer: Yes. So 9 January 2001, that line there is dark –
Charles Cipione: It’s a bit darker. So for up to the end of 2000 there’s the possibility of a new entry into the open balance. By the time we get to the beginning of 2001, I’ve run out of my source material. But what I wanted to show on this chart was that there were still open PEAKs or PinICLs that were sourced from 1996 to 2000 that still hadn’t been resolved.
So I’m running the chart out until when all of these PEAKS and PinICLs are resolved. But the open balance is probably, or is absolutely most pertinent from the end of ‘96 through the end of 2000 on this chart, where it’s just the run-out after 2001 begins.
Mr Beer: The graph shows a downward trend leading to a minimum number at that time in July ‘99 and then a very rapid rise; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: And you describe in your report that cresting in May 2000?
Charles Cipione: Yes.
Mr Beer: What conclusions overall did you draw from that analysis of the open balance of PinICLs and PEAKs by day?
Charles Cipione: Well, the first thing is there seems to be a lot of them. You know, everything’s relative on different systems, you know, what allows a PinICL or PEAK to be created, at what level of granularity is it. But having read it, there seems to be a lot of errors being raised regarding the Horizon System. So that’s just an overall comment about the chart.
You will notice that it looks as though, you know, in the middle of August of ‘99 the balance seems to be dropping. But as we’re approaching or as we’re getting to the end of August, all the way, you know, through looks like, well into the next year, there is a significant increase in the population of open errors in the system that need to be resolved.
Now that could be because that happens to concur with the national roll-out period, so just the volume of counters that exist, absolutely will have an input to this. But it also indicates to me that there are still significant issues being raised about the efficacy of the Horizon System.
Mr Beer: You conducted an analysis of the time taken fully to close a PinICL or PEAK, and that table is shown over the page, please. Can you explain what figure 16.2 shows.
Charles Cipione: So this is trying to represent how long from the inception of a PinICL or PEAK to its resolution. It’s basically showing its age. You know, what are the age categories of the PinICLs and PEAKs that were available for me to review?
The first bar chart indicates that, you know, more than 30,000 of them took from zero to 20 days to resolve, from opening to closing.
The next bar is 20 to 40, and it goes on, you know, from there. But there is – you know, at the end of the chart there is a box that represents everything that took more than 180 days to resolve. So this is just showing a distribution on how long it to look to resolve a particular PinICL or PEAK.
Mr Beer: So that last chunk at the end there, more than six months to resolve, does that equate to some 3,000 PinICLs and PEAKs requiring more than six months to resolve?
Charles Cipione: Yes.
Mr Beer: Did you form a view or draw any conclusions in relation to this analysis?
Charles Cipione: It appears to me that these take a long time to resolve. The amount of – basically the ageing of these PinICLs and PEAKs seemed to be very long, in my opinion, on how to resolve them. Of course every one is unique. There could be a particular issue that is outside of their control that they didn’t feel comfortable in actually officially closing a PinICL or PEAK down. That’s fine for one or two. But what I’m getting from this chart is that it takes a lot of time to actually close these down. So that doesn’t – it’s not a good look.
Mr Beer: You also, I think, analysed those people, I think all within ICL, who were involved in resolving 1,000 PEAKs or PinICLs or more, and there was a list of 48 of them, I think, and this is a sort of a league table; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: Barbara Longley, the first amongst them, an employee of ICL Pathway; is that right?
Charles Cipione: That’s my understanding, yes.
Mr Beer: Can we turn to section 17 of the report, please.
Sir Wyn Williams: Mr Beer, before we do that, can I just go back to the table at 16.1, just either to confirm what’s in my head or for Mr Cipione to dispel it.
He looked at the opening of PPs up until the end of 2000. So I’ve taken from that, Mr Cipione, that if we look at the chart which is more or less opposite 9 January 2001, those are the open PEAKs at that point, and it actually takes until, well, between September and December 2002, which is getting close to two years for some of them to be resolved. Have I got that right?
Charles Cipione: You have that right, sir.
Sir Wyn Williams: Fine, thank you.
Mr Beer: Can we turn to section 17 of your report, please, on page 118. Again the topic disclosed by the heading:
“Acceptance Incidents [AIs] were a gating issue to the financial success of ICL Pathway. A persisting issue related to AI376.”
Again to introduce this topic, can you explain what you’re addressing in this section of your report, please.
Charles Cipione: Yes. So my understanding from reading the material was that the acceptance was a term that POCL and ICL Pathway used to confirm that the system was good enough to be put into practice. It was good enough to be deployed and, more specifically, it actually triggered, I believe, the first payment to ICL Pathway for constructing the system and for implementing the system.
Mr Beer: You tell us in 17.1.2 that:
“Acceptance was financially significant to ICL Pathway. [Because] ICL Pathway was paid once acceptance was achieved. [And] it received a high degree of attention by ICL Pathway.”
The “it” there, are you referring to acceptance or the financial significance attributed to acceptance or –
Charles Cipione: Acceptance. I’m referring to acceptance.
Mr Beer: You tell us that the monthly reports describe outstanding or existing AIs, and ICL Pathway’s efforts to resolve them, and 376 caught your attention. It concerned accounting integrity. Why did AI376 catch your attention?
Charles Cipione: The Horizon System, while it is not the actual official financial accounting system for the Post Office, it is the source of all information for the financial – for the accounting for the Post Office. If there are any issues with accounting integrity that are sourcing from the Horizon System, that seems like – I mean, that’s the purpose – you know, that’s one of the main purposes of the Horizon System, to make sure that it is providing accounting data with integrity to the Post Office.
So when I saw that, that caught my attention, that that was a term associated with AI376.
Mr Beer: You point out that 376 was one of the final AIs to be closed.
Charles Cipione: That is correct.
Mr Beer: In terms of the chronology, you tell us that:
“24 September 1999 marked the day that Acceptance was granted, triggering [an invoice for payment of] £68 million … to be paid within 30 days.”
Then:
“In November ‘99, at least one full month and possibly two full months after acceptance …, ICL reported that ‘POCL have come round to the understanding that dealing with residual AI376 concerns in the short to medium term will rely on processes and tools but no new software features as such’.”
Why did that catch your attention?
Charles Cipione: I keep going back to, you know, the integrity of the accounting data that is being sourced from the Horizon System. That’s an extremely fundamental concept, and that is probably the most important item that – most important feature that the Horizon System should deliver.
The fact that this is still an issue troubles me when I read about it.
Mr Beer: Why does it trouble you?
Charles Cipione: Because, if that data doesn’t have integrity, the system is not performing it’s proper function.
Mr Beer: In 17.1.7, if we scroll down, please, you record that:
“In February 2000, ICL Pathway declared that the POCL Acceptance Manager has left the project and transferred the residual actions to ‘business-as-usual’”.
It was unclear to you exactly what took place to close AI376.
You say:
“The reading of these entries leaves much room for interpretation.”
What did you mean by that, please?
Charles Cipione: I was not clear from the materials that were provided to me exactly how this acceptance issue was closed. I know it’s a big deal. I know it’s a big deal. I keep seeing, I keep reading about it’s a big deal, and then it’s closed without a lot of commentary on exactly: was the system fixed? Is it perfect now? I did not derive anything from the material I was reading as to what happened to actually allow that particular Acceptance Incident to be closed.
Mr Beer: You just asked yourself a question there or posed a question. “Is it perfect now?” Is that what you would have required for accounting integrity?
Charles Cipione: I certainly would have required that I thought it was perfect. I mean, there’s no perfect system. There’s always the opportunity to introduce errors in any computer system. So perfect is not an achievable goal for any system.
However, you should think that it’s pretty perfect. You should think that there’s no reason that is out there that you will be introducing errors into your financial system, based off of how the technology is working.
Mr Beer: You then set out in 17.9.1:
“Regardless, the fact that accounting integrity was a persistent issue in the national roll-out of Horizon cannot have been the intention of the sponsors nor the goal of ICL Pathway.”
That might be stating the obvious. It’s plainly not a goal.
Charles Cipione: Of course not. I’m positive that ICL Pathway did not have that as their intention. I’m positive that Post Office, as the sponsor of the system, was not expecting that. Perhaps I wasted a few lines by writing that, but I just wanted to confirm that I don’t think that this was an intended feature of the system to have problems with the accounting integrity.
Mr Beer: But your focus has been instead on what was done to resolve it.
Charles Cipione: Yes.
Mr Beer: Is the state of your conclusion that you’re unsure exactly what was done to resolve the issue on AI376?
Charles Cipione: From this report, from the materials I was provided for this report, I could not figure out what was actually done to close that incident.
Mr Beer: Have you read or heard evidence since then which goes to that issue?
Charles Cipione: Yes.
Mr Beer: What was done; what is your understanding of what was done?
Charles Cipione: My understanding of what was done was that the benchmark for closing it continued to change throughout this period. There were a lot of tools and techniques and mechanisms that were required to be wrapped around the Horizon System to check for errors that the Horizon System might have been passing along through its processing. So I guess, when I read about that – so let me explain.
You have the source of information which is Horizon, which is supposed to be creating the accounting information that’s fed into the financial systems. We know through the PinICLs and through this AI376 that that was not operating correctly, at least during this period. So one option, which I think would have been the best option, was to go and make sure that the actual Horizon System was transacting correctly, was processing the data correctly, so that it had high integrity.
What I understand, from reading from the materials that I have been provided with subsequent to writing my report, is that there was a lot of difficulty in figuring out exactly what was introducing these errors in the Horizon system. So as a plan B, there were a lot of processes and procedures and tools that were trying to ring-fence the Horizon systems, which was known to be generating information that did not have the highest level of integrity, to try to catch those errors before it hit the Post Office’s financial systems.
Mr Beer: What was your view as to the introduction of a tool or tools that sought to recognise that there was a lack of financial integrity or accounting integrity?
Charles Cipione: Having a safety net around any financial system is fine. It’s always a good practice.
However, if you know that the source of information that’s creating the errors is your own system, that being ICL, I think that’s where the effort – there should have been a lot of effort in making sure that that was rock solid, that everything was being done to make sure that that system functioned exactly the way it needed to function, because that’s one of the basic features of this system. It’s a very important feature for any financial system.
You have to have integrity with the data, and I think, from the materials that I’ve read, it sounds like they were going away – at least during this period – going away from that particular effort to trying to just make sure that they knew errors were being generated, let’s have a plan B, a plan C, a plan D, to try to make sure that, if an error existed, it was caught before it actually got into the Post Office’s financial records.
Mr Beer: So, rather than putting effort into or all effort into identifying root causes, you are focusing on catching the consequences?
Charles Cipione: Yes.
Mr Beer: As a process what’s your view of that?
Charles Cipione: I think that having those type of controls is always a good practice, provided that you’re really trying to make sure that the genesis of the information is correct, has efficacy to it. If you’re relying on your safety nets to protect you, I would suggest that that’s poor practice. You need to be – sorry, if that’s one of your main reliances, that’s poor practice. You should always rely on your safety net. Safety nets are there for a reason. It’s always a good thing to have a safety net, or guard rails, or even a couple of layers of safety nets or guard rails. But if you are – from the materials I’ve read, it sounds like this was still an issue. This was an issue with the Horizon System.
If you continue to allow those safety nets to cover functionality that should be present in the core system, I think that’s a terrible practice, rather than correcting the core system in a way that you know will persist.
Mr Beer: Did you form a view whether the issues relating to AI376 were sufficiently serious to be bound up in whether the system should have been accepted or not?
Charles Cipione: Accounting integrity, absolutely. I’d say that’s probably the number 1 thing for this system, the accounting integrity.
Mr Beer: And, therefore, what was your view as to whether or not the issues with AI376 were, on the face of it, sufficiently serious as to affect acceptance?
Charles Cipione: It absolutely should have affected acceptance.
Mr Beer: Can we turn, please, to page 129 of your report. This is skipping over extracts from monthly reports concerning AI376 in particular.
At the foot of the page, you say you:
“Surveyed the PinICLs and PEAKs for the pattern of ‘Acceptance Incident’ followed by numeric or ‘AI’ followed by a numeric to identify PinICLs and PEAKs that dealt with Acceptance Incident issues. The following figure shows that 358 PinICLs and PEAKs were related to Acceptance Incidents and 223 PinICLs and PEAKs were specifically associated with AI376.”
If we go over the page, please, I think the lighter colour green relates to AI376; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: It’s slightly difficult to tell. Does that show that they persisted into after June 2000?
Charles Cipione: I also am having a hard time seeing the colours, but it looks like they are gone – it looks like the last entry for those are July of 2000.
Mr Beer: Thank you. Is there anything else you wish to say about Acceptance Incidents and in particular AI376?
Charles Cipione: The accounting integrity of the Horizon System, I think, is the most important part of the Horizon System. It is the reason – you know, beyond any – just the operational efficiency of the branches being worked, sure, that’s important, but no-one would ask for a point-of-sales system that they knew were connected to your financials if there was any doubt about the integrity of the accounting information that was being derived from it.
Mr Beer: Can we turn to section 18 of your report, please. Thank you. With the heading:
“Payment and receipt imbalances were common symptoms with varied causes.”
You say that what you previously discussed directed you to identify examples of accounting issues within the PinICLs and PEAKs, and in this part of your report you explore selected examples of accounting issues as represented by payment and receipt imbalance issues. Just describe to us why you are focusing on this, please.
Charles Cipione: I thought it was important to try to, number 1, support the last, you know, the prior theme that we just discussed. But I also thought it was important to illustrate how these issues were manifesting themselves. I think as we get to talk through the examples that I’ve put in it’s going to show that it wasn’t as there was a particular section of the Horizon System that was causing integrity issues as far as, like, the development code was concerned.
The causes were all over the place, in my opinion, as to what created these payment and receipt imbalances, and I thought it was important to just show that there were a variety of reasons that would illustrate how things could be out of balance.
Mr Beer: You remind us in 18.1.2 of the Cash Accounting Period 14 to 15 and then into 16, example that you gave earlier in your report which was, I think, an example showing how it should work.
Charles Cipione: Yes.
Mr Beer: You have updated that to show what happens when you introduce a bug into the system which causes the brought-forward balance to be incorrectly calculated. So, if we just scroll down, please, and look at fig 18.1, thank you. Can you talk us through it, please.
Charles Cipione: Certainly. So what I’m showing here is that, if the Horizon System was to inaccurately calculate the opening balance for cash and stock, in this example, just multiplying those values by three, as this particular receipts and payments account was reported out, it would show an imbalance between the receipts and the payments. So in this instance there’s, you know, approximately £11,000 difference or discrepancy on this particular report.
So this is just an illustration. This is something that I thought would be helpful for the chair just as an example.
Mr Beer: So this shows an imbalance of £11,000 where payments are greater than receipts?
Charles Cipione: Receipts would be greater than payments, I believe. Maybe I wrote that wrong.
Mr Beer: I was going from over the page –
Charles Cipione: Sorry, yes.
Mr Beer: Let’s just go back to the table and check.
Charles Cipione: I think that I might have switched receipts and payments here. But the important part is there’s an imbalance here.
Mr Beer: So this would show as, what, a surplus of £11,000?
Charles Cipione: Yes.
Mr Beer: To what extent could you ascertain whether that information that’s shown on this page here was visible, assuming this was a real-life example to a subpostmaster?
Charles Cipione: I will say that, during the course of my review this, in particular, was a little bit difficult to tease out from the information that I had. I did have some technical manuals, I did have some documentation available to me. None of the reports that were available in those manuals as representative of the reports that the branches might see looked as straightforward and clean as this. So I thought it was, just for the concepts, important to put this – I guess this is the way I would have shown it, this is to show the concept, but this certainly this not, I believe, anything close to what the people at the counters would have seen.
Mr Beer: If we go over the page, please, and look and 18.1.5, you say there are various other issues that could result in am imbalance: payments that were not recorded in Horizon; payments that were erroneously recorded in Horizon; receipts not recorded or erroneously recorded in Horizon; carried forward balance incorrect because the cash and/or stock were not correctly declared by the subpostmaster, or there have been cash and stock changes that couldn’t be accounted for.
You then describe a methodology. Can you explain what you did then, please, in relation to this issue.
Charles Cipione: Certainly. So in order to come up with the examples that eventually exist in this particular section, we wanted to basically search all of the PEAKs and PinICLs. Effectively what we did was we went through a number of iterations, looking at one or two trying to, manually, trying to understand the way they would look, and then start feeding in concepts or words into the Brainspace technology that I described at the very outset of this hearing.
Mr Beer: Are they described in figure 18.1?
Charles Cipione: Yes, yes. So, for instance, in this we’re looking for, you know, the concept of error, cash, issue, fail, and others. So what we’re looking for – basically what’s happening is Brainspace is taking entirety of the PEAKs and the PinICLs, and coming up with a bunch of words and phrases that are available for searching on. What this is showing is, you know, some what Brainspace calls a concept which is exactly how it sounds. What does it think – just without being told anything, what concepts could exist within this body of documents? One of them was error, one of them was issue, one of them was cash, and one of them was fail, that we have in the light blue highlighted here. So that’s the start of the process.
Mr Beer: So that gave you your first run of data, namely 38,803 PinICLs or PEAKs?
Charles Cipione: Yes.
Mr Beer: You explain that only a small number of these were rated highly relevant to each of the displayed concepts. Just explain what you mean by that.
Charles Cipione: So within these concepts, so as the Brainspace technology runs and includes different numbers of documents underneath each of these concepts, it has a grading for how closely aligned to the concept is this PinICL or PEAK and how weak aligned is it. What I’m saying here is that, although we have a large population, since this is kind of the outset of the process, as expected, not a high percentage of any one of the populations for each one of these concepts was considered very tightly aligned to that concept. So we needed to go on to another step.
Mr Beer: You explain that in 18.1.18:
“A review of those that top of the distribution revealed an issue mentioned in several PinICLs and PEAKs namely cash or stock not balancing. The common phrase being use was ‘receipts vs payments’.”
So did you use that concept for a new search?
Charles Cipione: Yes.
Mr Beer: Then, if we go over the page, please, what was the result of that new focused search?
Charles Cipione: So that provided a very focused set of documents. I believe it was 67 documents that was the result of the population underneath the receipts versus payments concept.
Mr Beer: Can you explain what 18.1.9 means, please.
Charles Cipione: Okay. So this was just another phrase, and it looks to be a technical term that was also associated here within the search that happened – it’s basically – it’s something that came up to be considered, to be looked at, and it looks as though it was – so it says:
“It’s worth noting that ‘EnteredBBF’ is commonly found in PPs when pasting in error messages from a manual migration message store.”
So from my understanding of the readings of the PEAKs and PinICLs, oftentimes the SSC will try to pull in information from the message stores, and they’re also entering – they’re trying to document the process that they’re using to fix it. When they happen to paste in information from the message store, this particular phrase happened to have been included in that information. So it shows up a lot. That’s all we’re saying.
Mr Beer: So the search returned 67 documents, and you actually read those?
Charles Cipione: Yes.
Mr Beer: You say that a CMML model was created. What does that mean?
Charles Cipione: I’m going to have to refer back to even refresh my memory of what the actual words are. But essentially what that means is we’re going through an iterative process because this is a machine-learning mechanism. The first step that we talked about was an unsupervised machine-learning process, but every step beyond that is what I’m going to be describing as a supervised machine-learning process.
What does that mean? That means that we are giving a particular problem to Brainspace to solve. We are allowing Brainspace to provide what it thinks the best answer is, and then we’re grading that answer and feeding the results back. So when I say grading that answer, so, for instance, I might say I want to look for a particular concept like error – you know, that was that beginning of this process – and let’s pretend that there was a document population of 100, and Brainspace came back and said, “I think 15 of these documents are related to the concept of error”. It’s now incumbent upon me as a reviewer to grade Brainspace.
So I might say, of those – what did I say, 11/15 – of the 15 that Brainspace thought was an error concept, I will say, “I agree with you on 11 of them and I disagree with you on 4 of them”, and feed that back into Brainspace and let it do the process again.
So now it not only has an idea of what I think is an error, it also has an idea of what I think is not an error, and it can then basically incrementally improve its search to align with what I think the concept of an error is, as opposed to what it thinks the concept of error is.
Mr Beer: Thank you. You say that three additional rounds of training were undertaken, as set out in your table. Just summarise what the additional rounds of training were for.
Charles Cipione: The additional rounds of training were the process I was just talking about, except for it was more than once. We kept doing it over. We kept telling Brainspace what we wanted, we’d get it back, we would grade and we would do it again, and get it back, and do it again, and get it back, until we get to the end process here which was, I believe, a 386 document.
Mr Beer: Then scroll down, please. I don’t think we need to address 1.12 or 1.13, do we?
Charles Cipione: No, no, they’re just technical descriptions.
Mr Beer: Therefore, if we come to your analysis, which is divided between quantitative and qualitative, starting with the former, can you explain what you’re describing in 18.1.4, please.
Charles Cipione: Certainly. So the result of the prior process was that we had identified 399 PPs. 137 were selected for review, and 127 as relevant. So that just sets –
Mr Beer: Relevant to the concept, the idea of the payment and receipt imbalance?
Charles Cipione: Exactly. So for that population the first thing we wanted to do was just to quantitatively describe those PPs, those PinICLs and PEAKs, across two of the data elements that are associated with those. One is the response category and one is the defect cause, and I think we’ve discussed those already today but I’d be happy to discuss them again if you would –
Mr Beer: If you would, just by reference starting with response category to the table at [18.16], please.
Charles Cipione: So as a PinICL or PEAK is processed, they start tracking what’s happening with this. We’re going to identify – you know, we’re going to put different values in what’s called the response category, which is basically how each individual that’s reviewing the PEAK or PinICL is describing it. This is the category – this is me as a review, this is my response to the categorisation of what I think this particular – the stage of this particular PEAK or PinICL is. So that’s one dimension.
The defect cause, that’s a little bit more straightforward. As I said before, that can change throughout the process, but ultimately there is a final value associated with the defect cause which is supposed to represent what the SSC thought the actual defect was for this particular PEAK or PinICL.
Mr Beer: In terms of the response category, I think you concluded that 19 per cent of the closure reasons, those which are described as “Other” in figure 18.3 and “Administrative Response” in 18.3, didn’t provide much or any insight into the investigation process; is that right?
Charles Cipione: That is correct.
Mr Beer: Just explain what the significant of that is.
Charles Cipione: So, you know, these data elements should provide a little bit of information about exactly how these PEAKs and PinICLs were resolved, you know, what the response was. When I see something like “Administrative”, it almost sounds like an “other” or catch-all. It’s just not very in informative to me as to what the issue is.
So, you know, for instance, in the chart in figure 18.3, one of them is “no fault in product”. That tells me a lot more than “administrative” response. At least I understand what the point of view was from the SSC on that particular PEAK or PinICL.
Mr Beer: Thank you. Over the page, please, and scroll down. Second element of the analysis, analysis of defect cause, and I think from this you found that a significant proportion of these PinICLs and PEAKs had defect causes that were recognised by ICL Pathway as being related to the design or development of Horizon. That is 45 per cent on the pie chart. Tell us which elements take you to the 45 per cent.
Charles Cipione: Certainly, and it’s in the footnote. So it’s, if the defect cause is design, design/high-level design, development/code, development/low-level design, development/reference data.
Mr Beer: That takes you to your 45.
Charles Cipione: Yes.
Mr Beer: You say in the narrative:
“This indicates [to you] that there were acknowledged bugs, errors or defects in Horizon that were capable of giving rise to a payment and receipt imbalance.”
Charles Cipione: Yes.
Mr Beer: Why did that indicate that to you?
Charles Cipione: Because the corrective – because the cause – identified by the SSC was it was a development or a design aspect of Horizon.
Mr Beer: So it’s self-describing?
Charles Cipione: Yes.
Mr Beer: What’s the consequence of that? What’s the significance of that?
Charles Cipione: The significance is this was an error in the Horizon System. That’s what was causing these imbalances.
Mr Beer: But also an error in the Horizon System giving rise to or capable of giving rise to a receipt or payment imbalance?
Charles Cipione: Absolutely, yes. So these are all receipt and payment issues that we’re talking about now. So everything we’re – everything that we’re going to discuss in this section has to do with that concept.
Mr Beer: You turning to analyse the time that it took to resolve PinICLs and PEAKs in this category. We can turn over the page to 18.5, and you tell us in the text underneath that only 26 or 20 per cent of them were fully closed within a week; 55, 43 per cent took five weeks or more fully to close; and 37, i.e. 29 per cent, took nine weeks or more fully to close.
Charles Cipione: Yes.
Mr Beer: What, if any, conclusion or observation did you make on that material?
Charles Cipione: Well, the best case scenario on this – let’s talk about the nine weeks or more. Let’s talk about the errors or defects that relate to those 37. The best case is that, after the initial error was identified and entered into the PEAK or PinICL system, there was another nine weeks where that error could have been manifesting itself throughout system. That’s the best case scenario unless something outside of this process was occurring that mitigated that issue.
Mr Beer: Can we turn over the page, please, to your qualitative observations. You say in 18.2.2 that it’s useful to examine individual PinICLs and PEAKs. You selected seven that highlight the varied causes of payment and receipt imbalances. How did you select them?
Charles Cipione: I picked them from the population that we were just discussing. So I picked them for purposes to illustrate the various possible causes of this particular symptom.
Mr Beer: So to give a fair cross-section of illustrative causes?
Charles Cipione: Yes.
Mr Beer: You summarise at a high level the PinICL or PEAK. You took some excerpts from the PEAK or PinICL and then you gave observations. We can see that in practice, if we go over the page, please, to look at the first of the seven examples.
Summary at the top, then the Chronology, and this isn’t all of what’s in the, in this case, PinICL. We’ve seen that some of them are very long indeed, running to many, many pages.
Charles Cipione: Yes.
Mr Beer: But you have taken that important parts of the chronology; is that right?
Charles Cipione: Yes.
Mr Beer: And then over the page, please:
“My observations”.
If you scroll down, please, you have asked a series of six questions which I think are the same repeating through the seven examples, and we can see what those questions are.
So that was the way that you were structuring your examination of the seven examples.
Charles Cipione: Yes.
Mr Beer: If we go back to page 137, please, at 18.2.4 you make the following general observations. Can you talk us through those five observations please.
Charles Cipione: Certainly. So many of these PEAKs and PinICLs seem to have been raised as a result of internal reconciliation. So that would have been what I was describing before, perhaps, you know, one of those safety nets that exists to detect errors. So these PinICLs or PEAKs weren’t necessarily raised by an SPM calling up and saying, “I have a problem.” They were more raised by the detective controls that had perhaps ringfenced the Horizon System.
Mr Beer: Thank you. The second observation?
Charles Cipione: So and I think I said this in one of my prior themes. Reading these, I think that the team that’s trying to resolve these is earnest in their effort to trying to fix these problems, at least as they saw the symptom. So I did not see any indication that they weren’t trying to solve them or trying to not do their job, and I think that’s important for the chair to know.
Mr Beer: Third observation.
Charles Cipione: Each one of the PEAKs and PinICLs had a lot of different teams that were involved. So oftentimes there would be – so there’s different development teams that are being – so there’s the SSC, and then there’s, you know, different development teams within the Horizon group that are working together to try to solve it.
So the SSC could try to solve something by themselves, but they might need to reach out to someone on the EPOSS development team or on the APS development team or on the LFS development team, depending on, you know, what time period you were in.
So this is just describing that there is a lot of conversation back and forth between people in the various groups.
Mr Beer: Then your fourth observation, you have told us about the earnest effort to investigate issues, identify a root cause and mitigate future recurrences. How successful was the earnest effort?
Charles Cipione: I think that the effort, the earnest effort that I was talking about, fixed something. It fixed something. But there’s – perhaps we can kind of step away from this particular example but just talk about what causes errors and at what level do you remediate errors.
There are symptoms, and there are probably lots of different levels of causes of those symptoms. The further up the food chain you can get to the cause, the more likely it is you are going to eliminate future occurrences of those particular symptoms.
What I’m describing here is: I know that something was fixed, for sure. I know that perhaps there was a correction entered into the system. So, for instance, there was a receipt and payment imbalance. Someone understands what that is and understands that at the Post Office that needs to be corrected. So perhaps one fix would be: I’m going to correct this particular instance of this symptom. That would probably be really just attacking the symptom.
Sometimes it might be: well, I know that there was a reference-data issue related to this, and I know that there was one specific part of the reference data issue reference-data system that was at fault here. So I’m going to go correct that, and also provide a correcting accounting entry to the Post Office.
But what if I know that the management of that reference data is really the root cause of that reference-data issue existing to begin with? There might be another level that we need to go to fix that and there might be another level about that.
So what I’m trying to describe here is: these were all addressed in an earnest way to actually resolve these particular PEAKs and PinICLs. I’m not clear on whether the appropriate activity to get at the ultimate genesis of all these problems was really ever addressed across the spectrum of all these, because the SSC has got a lot of work to do. We discussed that already in one of my themes. Their job is to fix this issue, and they did a good job on that.
What I’m not clear on is whether the true cause that’s creating all of these error instances is being addressed or not, and it’s just not apparent to me in each one of these very specific errors that are being addressed.
Mr Beer: That’s why you say it’s not evident that the identified issue was resolved?
Charles Cipione: Yes.
Mr Beer: So the symptom might have been cured?
Charles Cipione: Yes.
Mr Beer: Or the next proximate cause to the symptom might have been addressed?
Charles Cipione: Yes.
Mr Beer: But there wasn’t evidence that any root causes were addressed?
Charles Cipione: Well, I guess that really depends on what we define as the root cause. So we can have one degree of separation of root cause, two degrees. I mean, if we know that just one degree of separation actually is truly the root cause, there could have been things that were absolutely addressed. But I’m just not sure of what the degree of separation is here, because I’m only looking at the PinICLs and PEAKs information. I don’t know anything more the system for sure.
Mr Beer: Your fifth conclusion: in the majority of these PinICL and PEAKs, the root cause is related to Horizon. What do you mean by that? The riposte to that that might be: of course it’s related to Horizon. What are the other alternatives?
Charles Cipione: The first example we’re going to get to is not a Horizon-generated issue. So that one doesn’t have anything to do with Horizon; the rest of them do.
Mr Beer: Okay. What’s the significance of that?
Charles Cipione: The significance of that is these are not environmental factors that are causing these imbalance issues; it’s Horizon.
Mr Beer: It’s the system?
Charles Cipione: Yes.
Mr Beer: Can we turn then to the first example, please, on page 138 and 139. If we could display those together, please. It may be that we won’t be able to display all of them together and, therefore, at least in a readable way. We’ll do them page by page.
Thank you.
Can you talk us through this first example, please, concerning ECCO migration.
Charles Cipione: If it’s all right with you, Mr Beer, I am going to read it verbatim, each line and then –
Mr Beer: Then give an explanation?
Charles Cipione: Yes. All right. So the first line is:
“[The] system call related to TIP 1052. Please route this call to John Moran in EDSC.”
I believe EDSC is another term for the SSC. I think that’s correct; I’m not positive that’s correct. But what this is indicating to me is, related to TIP 1052. So I know that TIP is the Post Office’s accumulator of all the transactions for its accounting. That’s where Horizon is delivering its transactions to, at the Post Office.
So this would be an example of: since it’s coming from TIP this indicates to me that this is not coming from postmaster. This is coming from somewhere down dream saying that there’s a problem, okay.
Mr Beer: Yes.
Charles Cipione: All right. So, MSU, John Moran:
“The following has been copied from Business Call [I’ll skip that incident]. Discrepancy in Cash Account for week 46. A comparison between values received within the cash account files and those derived from the transaction stream … identified the following differences.”
So it’s showing differences. So what this is saying is: it looks like the cash account is showing different numbers than what the transactions that have already arrived in TIP are. So I’m imagining that there is a balancing that’s being done at the branch. It’s being sent up to the Post Office. The Post Office already has, in theory, all the transactions, and they are comparing what they think that should look like to what the branch is sending it, and there’s a discrepancy.
Mr Beer: There’s a shortfall of £97.39?
Charles Cipione: Yes. So now we’re asking the SSC to investigate. So that gets us to the next line.
All right, so we see some colour commentary here:
“This is an illustration of the stupidities that ECCO software allows.”
So just to refresh everyone’s memory, ECCO was a point-of-sales system that I believe existed in some branches prior to the Horizon system. So that’s why this is an example of an issue that is not really a Horizon issue, okay?
“A clerk can transfer cash and cheques between stock units without bothering to make sure that they match up. The result shows up when the transaction data is migrated to Horizon, which insists on clear demarcation between cash and cheques. In this case the critical transactions are Transfers In of two cheques whose total amount exactly equals the discrepancy noted.”
So what this is indicating to me is that ECCO allowed someone to call – let’s see – ECCO is allowing cash to be called a cheque. So that’s where they’re out of balance. The Post Office is thinking: I should have some cash and cheques, and Horizon is getting – this is probably an initial load of data into – this is probably an initial month in, at this particular branch, and when they migrated the data from ECCO to Horizon, ECCO incorrectly put everything into cheques instead of dividing it between cash and cheques. So that’s what the issue – that’s the source of this particular issue.
Then it goes on to say:
“I think Steve Warwick is aware of the ECCO problem here but as a matter of course I will route the call to him to allow him to comment.”
Steve Warwick says:
“This issue is well documented in previous incidents with TIP. The effect is that Pathway system reports the values of the affected products incorrectly on the Cash Account for the migration CAP although the cash account still balances. TIP then uses the Cash Account figures from the migration CAP as the start point for validating the next Cash Account received from the outlet and report a discrepancy between the transactions received in week 2 and Cash Account for week 2. This is a user error pre-migration of ECCO+ Office.”
This is not a Horizon issue. This is not a Horizon-generated issue. This is an issue that they’ve seen before when they are migrating from the ECCO system to the Horizon System. So then they do their passing back to MSU for issue of RED. I think that’s a reconciliation database. I can’t recall exactly what “RED” stands for but my understanding is that’s kind of a documentation method that the SSC uses to or Horizon ICL Pathway uses to show the Post Office that everything’s all right, and they issued it to the customer, no data error, close the call, and then they closed the call.
Mr Beer: Your observations, please.
Charles Cipione: So if I assume that the issuing of a RED notice to the customer resolved the PEAK or PinICL, then yes. So – and that’s my assumption but I just wanted to note that the resolution was that they documented it and sent it on to –
Mr Beer: So the immediate issue was fixed?
Charles Cipione: Yes, exactly.
Mr Beer: On that assumption?
Charles Cipione: Yes.
Mr Beer: Was a defect or root cause identified?
Charles Cipione: Yes, this was a migration – this was an ECCO-generated issue and it was identified.
Mr Beer: And was that correctly recorded? Was that defect or root cause correctly recorded in the PinICL?
Charles Cipione: I said yes on this. I think they indicated in the text here that this was because the user made that transfer, so I think that this is an appropriate identification.
Mr Beer: Next question: evidence that the defect or root cause was addressed. I think you said you wouldn’t expect it to be addressed?
Charles Cipione: I doubt it.
Mr Beer: Then more general observations; can you help us with those?
Charles Cipione: Right. So what I wanted to bring up here is that this was – this was generated through an internal reconciliation, which is fine – and this would have been one of the reasons why having that ring-fence around, you know, the system was good. This wasn’t an issue really that was generated from within Horizon, but it was an issue to make sure that the integrity of the data was right and that particular safety net worked correctly and appropriately in this case.
Mr Beer: And no response to an SPM was required. So he or she wouldn’t know about this fix?
Charles Cipione: That’s right. This wasn’t raised by an SPM; therefore, there was no need to call an SPM about this.
Mr Beer: So the “changes” in, in inverted commas, the SPMs’ data would be known without the SPM realising that there was a discrepancy or the cause of the discrepancy or the correction of the discrepancy?
Charles Cipione: That’s right. They would have been blissfully ignorant of this in an appropriate way.
Mr Beer: Yes. And then lastly observations on the defect root cause, you say that general user defect cause was correct?
Charles Cipione: Yes.
Mr Beer: Sir, that might be an appropriate moment to take the break before we move to example 2 of 7 because that will take a little while.
Sir Wyn Williams: Yes, of course.
Mr Beer: So might we say 1.55?
Sir Wyn Williams: Yes, fine.
Mr Beer: Thank you very much, sir.
(12.56 pm)
(Luncheon Adjournment)
(1.55 pm)
Mr Beer: Good afternoon, sir. Can you see and hear us?
Sir Wyn Williams: Yes, I can.
Mr Beer: Thank you very much.
Mr Cipione, can we look please at page 140 of your report, EXPG0000001, 140. We’re dealing here with the second of the seven exemplar PinICLs. Can we do as before, please; you talk us through.
Charles Cipione: Yes. As before I’m going to read and then explain.
Mr Beer: Translate, yes.
Charles Cipione: The first line:
“Incorrect CA value. Live trial the CA sub file for org units 12609 …”
Which is a particular “FAD”, which I understand to mean is a branch number identifier.
“… CA week 21 contains an entry for line 2050 with a value of £17,181.05. However, TIP has calculated from the transactions it has received that the value of the line should be £17,642.31. There is leaves a difference of £461.26.”
So there’s a problem. All right, so it moved on:
“Barbara, I have just spoken to John Pope (Requirements). This is classified under Acceptance Incident 376. Would you please raise the level of AI incident. Would John Simpkins please take a look, then send to EPOSS.”
Mr Beer: Just: “Would you please raise the level to an A/AI.”
Charles Cipione: Yes, “to an A.”
Mr Beer: Do you understand that to be the categorisation of A, B and C?
Charles Cipione: That is what I understand, where Category A would be the highest priority.
Mr Beer: Yes.
Charles Cipione: Right so John Simpkins responds:
“I have checked the agent boxes at Wigan for any T_HV_ALL event for this office between 12 August and 18 August and did not find any.”
Then Jim Anscomb says:
“There is a null value transaction Mode on [a particular number] for a cash credit of £143.22 though this is now not a problem for the harvester. No delays shown in the APR db.”
I don’t understand everything that’s in there except for what I’ve honed in on, this null transaction mode, which sounds to me as though there is a problem in the reference data for this transaction.
Mr Beer: Thank you.
Charles Cipione: All right. So the next line:
“The erroneous message was [a particular number] not [the number that the person before was referencing] – in case anyone else is relying on this [information]. We released a fix for this on 20 August into WP5406 [which I assume means work package, which is like a quick software release] which went to OTT and is due to be released in Tivoli package EPOSS_COUNTER_CORE~…”
This is very technical and I can’t explain everything, but it sounds like they are trying to release a software fix, or they are releasing a software fix, but it indicates it hasn’t made it to live, yet which means I don’t think it’s in production, meaning it’s not actually being used right now.
“The problem message is unfortunately an Existing Reversal message. So the harvester’s automatic assignment to Serve Customer is likely to provide problems, someone will need to amend this. Routing to EDSC for them to solve the procedural problems and check when the Tivoli package is due for release.”
So okay, the problem message is an existing reversal message. So it sounds like the regular process at the harvester would just be a step along the way of getting the information from the counter, all the way up through the data centres, then eventually on to Post Office. It sounds like the existing path – it sounds like they think it’s going to be a problem; you are not going to be able to do the regular thing.
Then we have Jim Anscomb saying the total discrepancy is 461.22 –
Mr Beer: 26?
Charles Cipione: 26, thank you. Indicates 143.22 has been accounted for above. Then it looks as though this next line is indicating that it’s a high priority call. It’s just indicating that, and with an encouragement to progress on this rapidly.
Then John Pope says:
“Just a thought, but the sign reversal mentioned above (serve customer sent to TIP instead of Existing Reversal) may explain 2 x 143.22 [because that’s] 286.44. Can anybody help with 174.82?”
So up to this point I think they’ve got some ideas. They’re trying to figure out what the problem is. I don’t think we’ve identified exactly what problem is as of yet, just in case anyone thinks that they have.
Mr Beer: Yes.
Charles Cipione: All right. So the next line Steve Warwick:
“It may be of interest that the value of the discrepancy between the TIP and Pathway figures appears to correspond to 2 x £230.63. During the balancing of stock unit AA on [18 August] a stock adjustment was made to reduce the value of Cheques by this amount, with a corresponding increase in Cash. These two stock adjustment records were later individually reversed, generating a further four transactions for £230.63, three against cash and one against cheques. Therefore, in total four cash transactions (two positive, two negative) and two cheque transactions (one positive and one negative) were written.
“Given that there have previously been issues with TIP’s rejection of ‘Existing Reversal’ transactions where the reversal settlement contained no cross-reference details, is it possible that this has caused the reconciliation failure? According to the message store data, the Cash Account for CAP 21 reported total receipts equal total payments, indicating that the message store data is complete and accurate.”
Okay, so here there’s some more speculation going on as well as an indication that in CAP 21 everything appears to be fine or, sorry, in CAP 21 the total receipts and total payments are equal. So Steve Warwick comes back again the next day:
“From further information received from TIP, the sequence of events seems to have been as follows …”
And I’m not going to read that time and date:
“A stock adjustments was carried out to reduce value of cheques by £230.63. This wrote two transactions – one to reduce the value of cheques, one to increase the value of cash by the same amount, both transactions carried mode ‘SAN’.”
Then:
“… a reversal of the cash settlement transaction to the cheque adjustment took place resulting in two transactions being written against cash, one to reduce the value of cash, one to increase the value of cash to settle the reversal. Both transactions carried the mode ‘ER’.”
And then:
“… a reversal of the cheque adjustment transaction was carried out, generating two transactions – one to increase the value of cheques and one to reduce the value of cash by the same amount.”
So he’s telling us what happened from his investigation:
“These transactions are recorded in the message store with the correct signs …”
Because that’s probably where he’s getting this information.”
“From the information supplied by TIP, it seems as though they have received/treated the transactions … (a reversal of a previous reduction in the value of cheques) as though it was a reduction in value rather than an increase in value, thereby calculating a discrepancy of twice that amount.”
So he’s still speculating on this last sentence:
“Either the sign on the transaction value sent to TIP was incorrect, or TIP have misinterpreted the data sent.”
So what this is telling me is they’ve identified where the difference is now. So that’s good. So they found at least what the symptom is, and now they are speculating on what happened in between to make those differences.
The next line:
“Looking at the TIP file there were two reversals for £230.63 in quick succession. The first is translated for TIP as balancing plus and minus entries. The second, however, is translated into two positive entries, which would account for the error. See extract of TIP file and message store attached.”
So what TIP received is not what TIP was intending to receive. So the next line:
“Changes to be made to clsEPOSS and clsTransaction in the EPOSSCore. Fix applied …”
So what they are saying here is they are changing code. To me this sounds like they are changing code modules.
Mr Beer: Sorry, they are changing code modules?
Charles Cipione: They are changing code, in particular, in the modules. I think that those are module names or the class names. I don’t know what the code looks like, so I’m speculating.
“You should get in the attribute grammar for a cash settlement for an ER transaction the additional data of …”
And it’s talking about the cross-reference mode.
“The harvesters need this.”
So this indicates to me that there was probably something wrong in the attribute grammar and, if you recall from our first session, that’s the format that the reference data –
Mr Beer: It’s written in?
Charles Cipione: Yes. Okay, so the next line:
“Testing of this should include transacting in each mode.”
So, if you recall, they will have been talking about two modes. One is ER mode and one is SAN mode. So that’s really more for them, just on the technical level, but those are the two modes that they are talking about.
“Testing should include transacting in each mode and the message should be as they were. Then performing a reversal of each mode an checking that the new attribute grammar exists in the cash settlements of the reversals.”
So they are just saying, when you test this, make sure you are using both modes and make sure you are doing a reversal, which is trying to recreate the set of circumstances that are associated with this error.
The next line:
“Link tested okay on CSR dev counter. Performed” –
Mr Beer: Just stopping there, do I understand it correctly that the problem that had been identified so far is with EPOSS?
Charles Cipione: That is the way I’m reading it, yes.
Mr Beer: Thank you. Sorry.
Charles Cipione: Okay.
“Link test OK on CSR dev counter. Performed a transaction followed by existing reversal for each of the following modes:
“Serve customer, Rems (all modes) [They list out the modes].
“On each existing reversal the message store was checked for the new attribute grammar.
“CrossReference.OMode – followed by the corresponding mode of the reversal.”
So the next line:
“WP_5766 has been applied to live. Routing call back to call logger for closure.”
So they think they’ve fixed it, they think they’ve identified the problem, made the changes and applied the fix. Then it says:
“We have seen that when a call is the subject of an Acceptance Incident (as this call is) then there is no point in us ringing up the originator to ask for closure. They always say that such calls are the subject of regular discussions between John Pope … and Martin Box … Eventually somebody at TIP rings us with a list of calls which can be closed. Accordingly I shall send this call to our holding stack to await such closure.”
Then it looks like it’s closed. So my inference from this is, since this is a high-priority issue for both the Post Office and Pathway, there’s already a mechanism going on behind the scenes to validate whether this can be closed or not.
Mr Beer: So just looking at the chronology then, this was opened on 20 August, and then closed on 30 December 1999?
Charles Cipione: Yes. I will say that – yes, I think it looks like they started waiting for – so it was officially closed on 30 December but I believe they think it’s been resolved –
Mr Beer: They being at the end of October?
Charles Cipione: Yes, exactly so just to be clear.
Mr Beer: Then your observations.
Charles Cipione: Okay. “So was the immediate issue fixed? Yes.” They fixed the issue and they received the proper closure instruction.
“Was the defect or root cause identified?” Yes, they identified the root cause. It was an issue being the signage and the grammar attributes sending the information to TIP.
“Was the defect root cause correctly recorded in the PinICL or PEAK?” It was recorded as code, development code, which we saw that they changed. So yes.
“Is there evidence that this defect root cause was addressed?” So I’ll just read what I said:
“The text indicates that a software update was made and tested, albeit the text only indicates that the testing occurred, not the results of this. However, I note that the text does not indicate that the test failed, which I would expect it to have said if that was the case.”
So I guess what I’m saying is: I saw the testing documentation here. It said it was ready to go live and they had a package. So I’m going to say: yes, it was addressed.
“Observations on the management and closure. The closure on the PinICL once the fix had been implemented and the original raiser’s approval was obtained seems appropriate to me.”
Then:
“Observations on the root cause. There is evidence in the ticket that the fix was implemented in LHITS to remediate …”
So I think that this was addressed. It seemed to be addressed appropriately. So that’s my observation.
Mr Beer: Thank you very much. Can we turn to the third example on page 144, “Reference data delivery issue”.
Charles Cipione: Sure.
Mr Beer: Again, if you can walk us through the text this one is much shorter.
Charles Cipione: “Office 91008 reports a difference in its receipt and payments totals for CAP 18. Please send this call to John Moran.”
So John Moran gets the call, and he indicates:
“I know what caused this problem. It was because reference data was not sent to the outlets concerning P&A products – the cash settlement was mapped to the cash account, but the corresponding transaction was not. If these transactions were recorded by in the Counter Transaction Exceptions report I could supply POCL TP with this information myself, but they have not been recorded.
“Can you supply the offending non-mapped transactions to this PinICL in message store extract so I can reconcile with Chesterfield.”
And then:
“The difference in the receipt and payments totals was caused by the fact that non-core reference data was not delivered to this office in time. Reference data was for OBCS products 177 to 185, and this reference data included primary mappings for these products. These products cannot be mapped to the cash account at stock unit rollover. That is what caused the difference in the receipts and payments total.
“This incident is related to 9 others all caused by this same problem. All the offices affected were migrated to Horizon on 20 July. All the offending transactions took place on 21 July when there was not reference data at the outlets. The correct reference data was delivered for business on 22 July.
“I have provided with the final BIM report an Excel spreadsheet (with the same file name as the BIM report) listing the offending transactions which were not mapped to the cash account.”
Then it goes on to say:
“Caller has raised BIM BASED ON THE EVIDENCE EXTRACTED AND so call has been closed.”
So this is the reference data for – non-core reference data, which my understanding is there were certain products that Post Office sold at every branch, and there were certain products that were sold only at some branches. Those products level into the non-core reference data.
Mr Beer: Over the page, please.
Charles Cipione: All right.
“Was the immediate issue fixed? Whilst the text states that the BIM was raised and the impacted transactions were identified and included with the BIM, it is not possible to determine from the PP [from the PinICL or PEAK] whether the transactions were updated to map them correctly. If I assume that the BIM process would rectify this issue, then it appears that the appropriate steps were taken …”
So I’m going to correct myself here on rereading this. I believe that the BIM process was more of a process to actually give correct transactions, not the process to update the reference data.
Mr Beer: I understand.
Charles Cipione: All right.
“Was the defect/root cause identified? The text indicates that this was a known issue and a clear root cause is provided (i.e., the product reference data required to correctly map the transactions to the cash account had not been delivered to … so the transactions were not mapped).”
“Was the defect root cause correctly recorded?”
It was recorded.
“The root cause recorded in this PEAK is “General-unknown”. The root cause is clearly defined in the PEAK and, therefore, the root cause is more akin to ‘Product reference data not delivered in time’. So that would seem like [a better description].
“Is there evidence that this defect/root cause was addressed? The text references the fact that the branches had now received the required reference data.”
So yes.
“Observations on the management and closure of the issue. This specific issue was fixed in that the reference data was delivered to the branch.
“Other observations. There’s evidence in the ticket that a fix was implemented in LHITS to remediate the identified issue. It’s unclear to me why the reference data was not timely delivered to the branch and nine others.”
So that would be kind of the next step up of, you know, what the root cause was. I mean, we know that the root cause of the specific transaction issue is because it wasn’t reference data there. We know that to fix that in the future the reference data should be there. So that’s one level of fixing, you know, of addressing the root cause.
But then the next level up of addressing the root cause is: why was that reference data late arriving? So they fixed it. I don’t know that they – I don’t know what was happening to fix the timeliness of all deliveries of reference data.
Mr Beer: Thank you. Can we turn to the fourth of the seven examples?
Charles Cipione: Certainly. This one is titled “Stock unit deletion”.
Mr Beer: On this one the text is relatively long.
Charles Cipione: Yes.
Mr Beer: Almost three pages. So could you pick out from the text – I’m not sure whether that’s possible, Mr Cipione, or not, the key entries?
Charles Cipione: Yes. Let me rest my voice while I look for the proper first citation. So the first entry just describing discrepancies.
Mr Beer: Just describe for us what the issue was.
Charles Cipione: So if we go, I think the meat of it is on page 147, the first entry for Steve Warwick. It’s the bottom of the middle of the page:
“The £155 error reported by TIP at FAD is almost certainly related to the MVL transaction. A number of these transactions took place in the week, and there was also a loss declared the previous week for this value against cash. The value of £155 was also transferred between two stock units during the week and a gain of £155 was recorded when balancing the end of CAP 18 (offsetting the loss of £155 declared at the end of CAP 17).
“Since there was no failure of the office to balance its cash account, it would seem that either one of these transactions had not been sent to TIP or TIP having miscalculated the value of the transaction.”
So that’s the beginning of the diagnosis.
Then at the bottom of that same page it says:
“The cause of this mis-balance … was the deletion of Stock Unit ZZ on 29 July before the [end of day] marker for the outlet had been written. This meant that the transaction carried out on the stock unit totalling £155 in CAP 19 was not reported to TIP.”
Mr Beer: Then maybe the Steve Warwick entry –
Charles Cipione: Four up from the bottom?
Mr Beer: Yes.
Charles Cipione: “The original response given to TIP was based on the fact that the symptom of the call appeared to be similar to other calls which had been identified as being a signing problem. This initial view was provided along with the statement that the incident was still under investigation and that once the evidence had been examined, the root cause would be determined. The root cause has now been determined and John Pope updated the spreadsheet shared with POCL regarding AI376. Closure will be agreed …”
And the root cause to TIP was the development code. I’m just referring back to the header of this particular sheet, so …
Mr Beer: Okay. Then your observations. You split these according to FADs. The FAD code is, I think, a numeric code which uniquely identifies a Post Office; is that right?
Charles Cipione: That is correct. That is my understanding anyway.
Mr Beer: So you collect together the first three –
Charles Cipione: Right. So this is a bit unusual, because it seems that there’s – I think this is one where there seems to be multiple – there seem to be multiple discrepancies included on one particular log file, which in theory – if they were all the same type of error, they could all be together. But, if there are different errors, they should have different PinICLs. But I think in this case there are two different issues going on here which we lost because I didn’t read every particular line. But just keep that in mind.
So for three of the FADs there was an RED notice to resolve the issue. So we think that that was fixed. For the FAD that we really read the meat about, for what we read, the reason we think it was fixed is because we know that David Salt, which was one of the items we didn’t read – someone wrote back and said that it was able to be closed. They agreed to closing the call. That would have been the third line up from the bottom.
Mr Beer: Yes.
Charles Cipione: “Root cause identified”. Okay. So –
Mr Beer: Over the page, please. Yes, the next page.
Charles Cipione: So for the first three there was an issue with mode attributes being null.
Mr Beer: What are mode attributes?
Charles Cipione: I believe that’s part of the information in the reference – information they are sending back. There was a piece of data missing when they were trying to send the information up, so it couldn’t be processed.
On the second set, the root cause is because the stock unit was deleted before the end-of-the-day marker had been written. So it was an ordering of operations issue, where they deleted a stock unit internally. This stock unit probably should have been deleted after the end of the day, or recognised – there is an order of operation issue that created the problem.
So was the defect/cause recorded correctly? The defect cause was recorded as development code, and I assume this applied really to the second item.
Mr Beer: So 523?
Charles Cipione: Yes. If this assumption isn’t correct, this seems like an appropriate defect code, but it’s unclear if this also applied to the other three which were a different issue. I’m not sure, besides the fact that they actually sent the RED in, which my understanding is that’s helping to correct the transactions. I’m not sure if they addressed what the actual problem was at least in this particular PEAK.
Observations on closure, here I’m noting that we’re talking about two different problems. So it’s a little difficult to follow, but it does appear as though they have addressed both problems, one with the RED and one with the development code.
Then ultimately on the root causes or, sorry, observation on defect root cause, for the first three I don’t have any information on what went on there. So I can’t comment on that other than I don’t have any information. I know that they are aware of it, and they sent corrections for the symptom, but I’m not sure what they did on the null mode.
The last one it’s identified – it seems appropriate to me based off of the reading of the text.
Mr Beer: Thank you. Can we turn to number 5, please, over the page, which concerns a brought forward balance multiplied by causing payments and receipt imbalance. This one is relatively long. I appreciate that asking you to summarise from your perspective is not the most helpful thing to do, because it means that you skip over text that’s important to explain or qualify your observations. But do take your time, if you just need to look or remind yourself of the problem or the report of the problem from the customer call. (Pause)
Charles Cipione: I think the first item that we should be looking at is at the top of page 151.
Mr Beer: Yes, “Initial investigations”?
Charles Cipione: “Initial investigations have shown that the problem arose at the time that the Office Trial Balance report was produced. On the Office Trial Balance report the brought forward value was £71,000 instead of £14,000. This appears to have been caused by the creation of a correctional stock unit which was additional to the normal stock unit. Due to an error in the code, when the stock unit balance records are read, the first stock unit (22 – first in alphabetical sequence) is correctly identified as having no ‘Brought Forward’ value from the previous week, the system then incorrectly assumes that this must be a migration week and generates a brought forward value for the stock unit which is incorrect. The error is being investigated for urgent correction.”
So this is a code error.
Mr Beer: I think you can probably see the response to it over the page on page 152, second entry from Steve Warwick.
Charles Cipione: “The doubling (or multiplying) of the brought roaring value on the cash account has been traced to an error in the changes which were delivered to correct the previous problem related to the previewing of the ‘Final’ cash account.”
So what I’m reading from this is there was a different error. A code correction was made for the error, but now it’s creating this particular error. So they have identified this error as being related to a previous code correction. So that’s effectively what the cause of this particular error is.
I’m happy to read the rest of it, but I understand you trying to be efficient with time. So I could skip over the rest of that but that’s the cause.
Mr Beer: Then go to page 153 then, your observation.
Charles Cipione: “Was the immediate issue fixed? I don’t see evidence in the PinICL or PEAK that suggests that the issue at the FAD was fixed although the text does not that the FAD completed their roll-over, presumably with the imbalance still in place. Perhaps the alert to the Horizon Helpdesk indicates that the corrective procedures were communicated …”
So what I’m saying here is I’m just not sure if it was actually fixed. It wasn’t clearly indicated in here, but it certainly doesn’t mean that it wasn’t fixed.
All right.
“Was the defect root cause identified? Yes, the root cause was an unforeseen consequence of the current LT1 fix”, which is what I just described.
“Was the defect root cause correctly recorded? [It] was recorded as development code. This seems appropriate since a software fix was required to [fix the problem].”
“Is there evidence that this defect/root cause was addressed? Yes …
“Observations on the closure. The root cause was identified, addressed in an upcoming LT2 release, and a workaround was communicated to the Helpdesk.”
Then that’s pretty much the same observation I had for the overall observation on the root cause and defect.
Mr Beer: Thank you. Penultimately example 6 on page 154, please, which is “Navigating to a different mode whilst transactions are on the stack.” Can you just explain what that means.
Charles Cipione: So what’s happening here is, if you remember from my last testimony, there’s the ability for the counter user to start adding things to the stack, which is basically the customer list. I have got a whole bunch of things that I want to perform for a customer. So what it looks like here is that they’re in the Serve Customer mode, they have started adding things to the stack, and then for some reason they’re wanting to switch a mode. I believe that that’s what is creating the issue; it’s changing from a Serve Customer mode to some other type of mode while there are items on the stack.
Mr Beer: You can see that this afflicted 28 offices from the first customer call on 5 November ‘99.
Charles Cipione: Right. It would appear that way from the beginning part of the text of this PinICL. But they have identified that they know that 24 of them were already – that these differences were already accepted at migration. So they were – 24 of them are known migration issues and they have already actually addressed that. So we’re left with four FADs that we need to deal with.
I will preview that I know that one of these FADs also is a migration issue. So it’s really three FADs that have this actual mode-change issue related to their problems.
Mr Beer: Then, if you go over the page, I think you can see how, by reference to the FAD code there, what the response was in relation to each of those three. In fact, there’s a fourth as well.
Charles Cipione: Yes. So the first one it says it’s a housekeeping transaction. It was carried out for a value and it was not settled in the user – so the first FAD, the third FAD and the fourth FAD in this list, they are all indicating that they had transactions on the stack, they weren’t settled, and the user at the branch navigated to a Revaluation Up menu, and that’s what caused the problem. For the second one it was just a discrepancy at migration. That’s the one I was indicating.
So those are the three that really had the issue.
Mr Beer: Can we go over the page then to your obligations.
Charles Cipione: Sure.
“Was the immediate issue fixed? For 25 of the 28 FADs the difference had been handled at the time of migration. For the remaining three, I don’t see explicit evidence in the PinICL that suggests that the issue with the FADs was fixed, but …”
And there was more text in here about POCL’s response or non-response.
“… but POCL’s exclusion of these FADs from their ‘list of call to remain open’ seems to indicate that the issue was resolved to their, [meaning POCL’s], satisfaction.”
So there must have been a mechanism between ICL Pathway and POCL for indicating: we know there’s an issue and then having an open-issues chart at the end, and these weren’t on it.
“Was the defect root cause identified? The root cause for 25 of the 28 related to initial migration. For the remainder, the root cause was identified, namely that Horizon allowed a user to navigate to a different mode while transactions were on the stack, which would then be subsequently settled incorrectly.
“The defect … was recorded as reference data.
I have here “Without a comprehensive understanding of which component of the system contained the error, it’s not possible for me to say whether this is an appropriate code”. I would say it seems like it could be an appropriate code, especially since I know or since I believe that the modes are governed by the reference data. So they put the reference data there.
“The PinICL states that the error was corrected in live software.” That’s for “Was it addressed?”
“This one was closed without positive confirmation that the imbalances were addressed”, but “the root cause appears to have been identified and remedied.”
Mr Beer: Lastly number, 7 please, page 157, PinICL concerned with a data tree build failure. I think you can see Customer Call, first entry, what the issue is.
Charles Cipione: Yes. So the cash account line comparisons – so let’s go:
“The office had big problems with its receipts and payments. CAP 5, 6 and 7 did not match.”
Then it lists the differences. And so we get down to the first substantial Steve Warwick entry:
“The cause of the problem in all 3 CAPs at this outlet was the fact that Stock Unit DD’s rollover records from CAP 5 and 6 represented a ‘nil balance’ (the total stock holding was nil, no receipts or payments transactions were recorded) despite the fact that that stock unit had been trading normally during the period. [Said] This issue was raised [earlier] and is still under investigation …”
Then it says:
“The fact that the Stock Unit DD’s transactions and stock holdings were omitted from CAP 5 Cash Account meant that the Brought Forward value for the office in CAP 6 was incorrect. This caused CAP 6 Cash Account to mis-balance. I am still investigating by CAP 7 cash account mis-balanced, but I note that the office returned to a balanced position in CAP 8.”
Mr Beer: Then I think if you go over the page.
Charles Cipione: Yes.
Mr Beer: And scroll down to John Moran’s last entry –
Charles Cipione: Which one would you like me to start at?
Mr Beer: I mean, this is more for you than me, but the 12 July 2000 entry.
Charles Cipione: Okay. So John Moran is asking:
“I need to know what correct Cash Account figures should have been were it not for the Dataserver failure.”
So in between our things they’ve identified a data server failure.
Mr Beer: Yes.
Charles Cipione: “Can these be derived from the transactions in the message store …”
So John is asking questions in order to help remedy this situation.
“2. The diagnostic code which was delivered before this incident happened was promised to aid in investigating the cause of the problem. Has this code helped? How?”
“At some point a work package was delivered would alert the user that there was a problem with the [stock unit] rollover and the user would be prompted with a message to redo the rollover. Has this been delivered? If so, why did the rollover process not cease and prompt the user to try again?”
So just a little colour. This indicates to me that they have seen this type of issue before, and they’ve tried to put in some safety procedures at the actual Horizon level to help the user make sure that they can’t generate this problem.
So I’m going to go down to the first entry on page 159.
Mr Beer: Thank you.
Charles Cipione: Just at the end, the final paragraph, not the three numbered:
“I don’t think I’m being premature in revealing that we think we know why these failures with the Dataserver or occurring. Steve Warwick experienced such a failure on a rig he was testing against and found the root cause was that the Archiving was active during a riposte query; this only occurs ‘out-of-hours’ at end of each working day. Archiving will occur ‘in-hours’ should the counter have been switched off overnight for seven consecutive days and hence the sporadic nature of these incidents.”
So what this is saying is there’s an unexpected issue when archiving is occurring, and there’s Riposte activity, and Riposte is the part of the system that’s helping manage the message store, which is where they are keeping all the transactions. It’s not – it seems to be that there’s some sort of conflict going on when archiving starts up, and they are trying to do this. So that’s what’s suspected to have actually caused this problem.
If we go to the middle of the page, they are describing here essentially that they are having problems trying to actually tease out the actual details to walk forward and everything. I don’t think we need to read it all, but they are saying that they are having problems looking at the actual detail to resurrect – because remember this a CAP 5, 6, 7 issue. They are having trouble figuring out how they could actually see everything that was going on in each one of those CAPs.
Then at the bottom it says:
“I’m not sure it’s worth spending time trying to resurrect the other CAPs. The method I have derived assumes that the CashAccLines for the previous CAP” – The cash lines, cash access lines. Cash account lines probably.
“… for the previous CAP. I see from Steve Warwick’s analysis that CAP 6 was not correct as well. Now, if I rerun the tool I have developed on CAP 6 it will use the base line in the Cash Account Line figures from CAP 5 which we know are wrong and I have just recalculated. I think therefore enough time has been spent on this problem and it’s not cost effective to proceed further. However, in future where the problem is just one CAP, we should be able to resurrect the figures more easily.”
So what they are saying is, since this was a multi-CAP problem, there’s no way for them to go back and resurrect information from CAPs that are more than – that are further than the most recent CAP.
Mr Beer: Your observations, please.
Charles Cipione: “Was the immediate issue fixed? It appears that CAP 5 account was reconstructed, but that CAP 6 and CAP 7 accounts weren’t.”
I don’t know if that’s really a problem or not, but it’s something that they acknowledged.
“Was the defect root cause identified? A root cause was identified, namely that the archiving process was active during the Riposte query.
“Was the defect cause correctly recorded?”
It said the development code was recorded, which I think is appropriate.
“Is there evidence that this defect root cause was addressed? The reference” – there was a release a reference to a release that suggested that it would address this particular issue, and something that we didn’t read, and then it appears that the investigation was carried out with – there was a lot of discussion.
So this would have been an example of: look, they are working towards finding at least an answer for this particular situation, and they think that this next release should take care of it. So that’s my observations.
Mr Beer: Thank you. That can come down from the screen now. So stepping back, having looked at those seven examples are, they evidence that payment and receipt imbalances were commonly reported?
Charles Cipione: Yes.
Mr Beer: Taken together with the wider dataset, the non-examples, things that we’ve not looked at, at examples, but which you found through your search methodology?
Charles Cipione: Yes.
Mr Beer: Are they evidence that the payment and receipt imbalances that were commonly reported had a variety of causes?
Charles Cipione: Oh, yes.
Mr Beer: Are they evidence that there was an earnest attempt by the SSC, in particular, to seek to resolve the issues raised?
Charles Cipione: Yes.
Mr Beer: But do they also evidence that the majority of PinICLs, six out of seven on the examples, all had as a root cause a problem with Horizon?
Charles Cipione: Some were within Horizon, yes.
Mr Beer: Have you got any overall observations to make in relation to this section of your report? What does your work on this tell you, if anything?
Charles Cipione: When I see evidence of issues where there seems to be a lot of different reasons for causing the same problem, number 1, that that obviously makes it harder for the people that are trying to track down to fix it because all they are working off of is: Here’s the symptom, you know. “Doctor, I’m tired. What’s wrong with me?” You know, that type of thing. Well, it could be a lot of things wrong with you. Let’s investigate a little bit more. But it takes that investigation process. So that’s one issue.
Second is, there seems to be a lot of different – at least at some level of separation, a lot of potential different causes for these problems, which is troubling, because you wouldn’t – especially for this type of issue, it just looks like the system has a lot of instability in it, and it’s not apparent to me looking at these PinICLs that it’s isolated in one place. Therefore it would probably be difficult to just focus on one little place to fix those instability issues.
Mr Beer: Thank you. Can we go back to sections 11 and 12 of your report which I skipped over. 11 is on page 89. Thank you. This section of your report is given over to the observation by you that financial concerns were considered by ICL Pathway throughout the time period reviewed. You observed that ICL Pathway is a profit-seeking entity and you therefore assume that it was motivated to deliver the system and to make a profit.
In the light of that observation, which is, if I may say so, a statement of the obvious, why have you included this in your report? What stood out for you, if anything?
Charles Cipione: So the materials I relied on for this analysis included the monthly reports, and there seemed to be a lot of discussion about financial issues, you know, costs, timelines, and how it might affect, or acceptance and how might affect payments. It seemed to be prevalent in the monthly reports, and I agree, it’s obvious. But those were the documents I was given, and I felt as though, if I didn’t say something to that effect, it might look like I’m not discharging my responsibility correctly.
Mr Beer: You say at 11.1.3 that:
“The financial success of Horizon relied on Pathway orchestrating many different constituencies: Sponsors, suppliers and their many internal groups. Benchmarks relating to acceptance, the timing of roll-out coverage, adherence to service level agreements requirements, were topics of discussion as they were directly connected to revenue, cost, and profit realisation.”
Is that always the case in an IT project?
Charles Cipione: Yes.
Mr Beer: “The balancing act between operational and technical activities and their financial ramifications was highlighted in the April 98 report: ‘Should the pressures mount, the temptation to hold to NR2 dates at all costs is immense. If we were to (purely theoretically) compromise NR2 quality in order to hold timescales, we would almost certainly be worse off in the long run.’”
“Acceptance was received achieve on 24 September ‘99, triggering the first invoice … roll-out coverage incentive for 1,800 outlets was achieved on 5 November 99. The first payment (£105 million) was received in early December ‘99.”
You say in 11.1.5:
“The items [you then include in the following table] illustrate some financial wins and some financial losses from ICL Pathway’s perspective. Financial concerns were weighed against the resource allocations to deliver the Horizon IT system. It’s my opinion that the financial aspects of delivering the Horizon IT System affected the decision-making process.”
Can you explain or expand upon that last sentence, please.
Charles Cipione: Yes. Since ICL is a profit-seeking entity, obviously they are wanting to make sure that this is a profitable endeavour. When they are discussing concepts such as making sure that they get acceptance, and how that relates to timelines, and then how time slippages are creating concerns with their particular partners, their vendors, all of that together just tells me that that’s obviously on their mind, which it should be of course and, like I said, this is stating the obvious, but I think I would be remiss not to comment on it, since there was quite bit of text in the monthly reports concerning it.
Mr Beer: To what extent, if any, did you ascribe Fujitsu’s financial concerns to the fact that, initially at least, the programme was being run and financed under a private finance initiative, and then one of the significant partners from which much revenue would be obtained withdrew?
Charles Cipione: Could you repeat that question, please.
Mr Beer: To what extent did you ascribe Fujitsu’s financial concerns to the fact that, at least initially, the programme was being run and financed under a private finance initiative from which one of the major partners withdrew?
Charles Cipione: I will say that, you know, as far as the private finance initiative is concerned, this is my first exposure to what even a private finance initiative is, and it probably didn’t really resonate with me that much, because I’m not sure that I actually really took that into account.
I think, now that I maybe understand that a little bit better, I think that, you know, that would explain why they didn’t get paid up until a particular acceptance point, just from the witness statements that I read after I submitted the paper. So that has helped contextualise the acceptance period.
I think I was more focusing on their not getting paid until they get acceptance, without really understanding the linkage between that particular event and the fact that it was originally a private finance initiative that they went into with another partner.
I also don’t know that I had much understanding of the impact of the BA’s withdrawal either. It wasn’t really on my radar. I didn’t understand the implications of it as much as I do now. So I would say it didn’t really impact it a lot.
Mr Beer: Thank you. That’s very fair. In which case I won’t ask you any further questions about that.
Some witnesses to the Inquiry have suggested in the course of their evidence that a consequences of this being initially a private finance initiative run-and-financed programme, and then it being a legacy of a private finance initiative was that the Post Office was not allowed to see the high or low-level design of the Horizon System, and was required instead to trust Pathway to produce acceptable outcomes.
In the design delivery of IT projects, have you encountered what has been described as a black-box approach such as that before?
Charles Cipione: Have I experienced it personally, no. I mean, I understand what it is, and I certainly can appreciate that sometimes that’s okay, but personally I’ve never engaged in a project where it’s been a completely black box.
Mr Beer: Thank you, in which case I won’t ask you questions about that either.
Can we turn to section 12 of your report, please.
You refer to “The tenuous relationship between ICL Pathway, their sponsors (initially the Benefits Agency and POCL) and suppliers were often topics of concern for ICL Pathway’s management team.” Can you describe overall what this section of your report is addressing.
Charles Cipione: Certainly.
Mr Beer: What was the issue for you?
Charles Cipione: The issue is this is a very complex system that ICL Pathway is trying to create and, notwithstanding what you just said, they’re not trying to create this in a vacuum. They need to have a good working relationship with their sponsors as well as with all of their co-suppliers of this system.
So, if there are problems in communications, either because of expectations being misaligned or for whatever reason, that’s going to cause problems. It’s just not a healthy working environment. Everyone needs to basically be rowing in the same direction in order to get something like this done on time and efficiently.
So overall my commentary around here is that it doesn’t – it seems that there were times where there’s a lot of tension between ICL Pathway and either its sponsors or potential tension between ICL Pathway and some of their partners in delivering the system, and the reason I know that is because, you know, it was talked about in the monthly reports. That just struck me as something that I’m sure everyone wishes wouldn’t have happened, but certainly couldn’t have provided the most healthy environment in making sure that the Horizon System was the sole focus on getting produced.
Mr Beer: So far as you could see, was this a consistent problem over the entirety of the period that you looked at, even after the Benefits Agency withdrew?
Charles Cipione: I don’t think it was a consistent problem over the entire period, but I think that there were points in time, based off of the information in the monthly reports where it was a concern.
Mr Beer: Thank you. Can we turn back, please, to page 4 of your report. This is your Executive Summary on pages 4 and 5 and I would like, if you would, for you to speak to these nine or so statements that you make. So the first:
“Horizon IT envisaged modernising the UK’s Post Offices branches. This was an ambitious goal placing hardware and software in about 18,000 branches to allow subpostmasters and mistresses the convenience reliably to store and transmit electronic records of their daily business activities.”
The words “this was an ambitious goal”, what did you intend to convey by that?
Charles Cipione: I wanted to employ language that would indicate the complexity of this roll-out of this system. It is ambitious. There are a lot of moving parts in this system, a lot of moving parts. In any system, especially a technology system, the more moving parts you have, the more opportunities there are for problems. There are a lot of managerial tasks both strategically and tactically that need to be tackled. There are a lot of logistical tasks. There are a lot of training tasks, and there’s a lot of technology tasks. The technology can be spread across infrastructure, hardware, software, any number of things.
The issue with computer systems is that, like I said, there are a lot of moving parts, and any one of those can cause a problem. So it’s ambitious to say that I’m going to make sure that all of these parts are co-ordinated in a way that it’s going to deliver exactly what you want me to deliver. So that’s why I think it’s ambitious.
Mr Beer: Thank you. Second:
“The technical aspects of the Horizon IT System were significant but did not account for all of the complexities of a successful implementation.”
What did you mean by that, please?
Charles Cipione: So the technology certainly is the focus and is there. But even aspects such as training – do my end users know how to use the system – that’s not necessarily a technical issue but it’s a very important part of the system with a capital S. You know, are we going to make this system work correctly?
Training, you know, even delivery of replacement hardware, not what I would think of as a technical issue, but logistically very important to the process. Are we making sure that this systems up all the time? Do we have good management communications amongst all of the different players that are here to either provide the requirements, implement those requirements through our design and development process, or support those requirements, like an infrastructure backbone such like BT or Energis making sure that everyone understands what’s going on?
So those are kind of the issues that are orbiting the fact that even just the technology itself is complex, there’s a lot more to this system than just sitting down and coding it. I would say that was the system with a small S, and making sure the whole thing works is the system with the capital S.
Mr Beer: You say a sentence on:
“Both the technical and organisational dimensions of the Horizon IT System also required vigilance. An IT system is a ‘living’ entity. It needs care and attention beyond its initial roll-out.”
Can you expand on that.
Charles Cipione: Yes, this is not a point in time thing. Let’s say that I’ve delivered to you the perfect system today. That doesn’t mean that you will be satisfied with it in six months, because perhaps you might want some changes to it, you might want to either expand the volume of activity that can be done on it, or you want some more functionality or features. So there’s going to be some sort of feedback loop process which will most likely require me to continue augmenting the system.
That’s another dimension that continues to complicate making sure that this system with a capital S works right because now, not only have I delivered you a system right now, we’re talking about evolving the system and having it mature over time, but making sure that it’s stable throughout that whole period.
That’s the living system and, like I said in my discussion before, it’s not just the technology. It’s the technology and the people and the processes that that technology support, all being co-ordinated and making sure that they are in concert with each other as opposed to being at odds with each other, and that’s difficult.
It just is. It’s difficult to have that many different items working together correctly. So does that answer your question?
Mr Beer: Yes. Could you help: does the level of care, attention and ongoing vigilance that one is required to give to an IT system, in part, depend on what has occurred in design, development, testing and roll-out, i.e. how easy a birth the product had?
Charles Cipione: Can you roll back and ask me the question again, please.
Mr Beer: To what extent does the level of care, attention or ongoing vigilance that one has to give over to an IT system depend in any way on what occurred in design, development, testing and roll-out?
Charles Cipione: I will say that, in a perfect world, the design phase of any system is very much a partnership between a sponsor of the system and the team that’s delivering the system, and they need to be very closely aligned and in very good communication, in order to produce a final product that satisfies the sponsor’s needs.
Having said that, oftentimes, you know, we can all of a sudden start seeing an introduction of new requirements in the design phase if we’re not disciplined about it. You know, there’s got to be a point in time when you say, “This is what we’re going to do and this is what we’re not going to do”, because we could be on an endless cycle of, “This also would be nice, this also would be nice, this also would be nice.”
So it’s very important, I believe, for the sponsors to have a really good idea of, hopefully, generally what they want and then be committed to specifically talk in detail of whether it’s okay to deliver it in this manner, or whether it’s okay to be delivered in this other manner.
That’s a discussion to be had. Unless the sponsors know exactly what they want at a very detailed level at the beginning, in which case they can say, “Here’s exactly what we want, go do it and you don’t need to talk to us until it’s done.” That’s quite rare, but – I mean, as a programmer, I would love that. You tell me – you know exactly what you want and don’t make a change to that, I would love that because then I just have to go and actually execute against that.
But that’s not typically what happens in the real world because usually discussion – you know, you didn’t know what was possible. I had a couple of comments about what you asked for, and it might change your mind about what you want. So there’s obviously a dialectic going on there, but that should be very short in time to the point where you say, “Okay, I know that there’s a wide range of possibilities available, but for right now let’s focus on what I want immediately. Let’s get that done, make sure it works right, and then we can move on to the next stage.”
Mr Beer: Thank you. Paragraph 1.1.3. You say:
“The Horizon IT System had multiple constituencies that needed to be both strategically and tactically aligned.”
What do you mean by that strategic and tactical alignment?
Charles Cipione: I think I was actually just referring to what I just said. So strategically that’s a very high level, right? We know we want – you know, in this case we know we want a network system that can do our point of sale and interface with our accounting system. Okay, great. That’s a good strategy, and there’s a business reason behind that strategy, and I’m more than happy to supply that to you if I was the supplier.
The tactical part though is where a lot of the heavy work is. We need to know specifically how to do things. We need to specifically understand what you’re expectations are.
For instance, you know, if we go back to my first testimony when I said, “Do you have a set of business processes that are mature and you want this system to support?” Okay, let’s talk about those, let’s make sure that the system that we’re developing supports those, and then we need to go back to our team, to my team and all of my partners, and I need to make sure that we all understand exactly what the needs of those business processes are and how we’re going to first architect a solution and then actually build and deliver that solution.
Mr Beer: Thank you. You say in point 4:
“The systems design and implementation needed to account for real-world contingencies. Many designs are very elegant, but they only maintain their elegance if the implementation withstands practical realities and concerns.”
I think that probably speaks for itself unless you need to expand on it.
Charles Cipione: No, I think that speaks for itself.
Mr Beer: Thank you.
1.5:
“The user support mechanism needed to assist the users as they migrated from a paper-based process to computer-based process or switch from using one system for another.”
That latter is a reference to –
Charles Cipione: ECCO.
Mr Beer: To the new EPOSS system:
“Continuous training for all versions of the system needed to be available. The support structure needed to cater for end-to-end users who might struggle to adapt to changes from the manual processes they had undertaken for many years. The support structure needed to be able to service a high volume of users. The support structure needed to be designed to adapt to the needs of the users as the IT system evolved through its different versions.”
I think you have spoken to that already and therefore we needn’t expand upon it.
You say, sixthly:
“The internal error resolution mechanism needed to be able to quickly resolve reported errors through identifying root causes, methodically correcting these errors and distributing the remedies in a timely and efficient manner.”
Was there any particular reason why that was necessary in Horizon as opposed to any IT system?
Charles Cipione: This is for certainly for any IT system. This is, you know, some of the material I was asked to review, so it definitely needed to have a mention.
Mr Beer: Seventh:
“The system’s functionality needed to maintain accounting integrity. It was the origin of sales and inventory information that flowed into the financial systems of POCL. Consequently, it was intended to be a ‘source of truth’ for these fundamental accounting facts. Any errors deriving from the IT system would, if not the otherwise rectified, be reflected in all downstream processes and systems.”
I think that probably speaks for itself.
You say in your paragraph 8 that:
“Throughout [your review you] identified shortcomings in each one of these key areas.”
The first of them is “constituency alignment”. Can you speak to each of these six points, please, the first, constituency alignment.
Charles Cipione: Yes. You know, through the reading of the documents, it was obvious to me, at least through my reading of the information, that there were some strained relationships between ICL Pathway and POCL. It was actually fairly straightforward, and that’s just not an ideal situation when you are trying to work together to develop a system.
Mr Beer: You say:
“The Helpdesk was often the root of Service Level Agreement issues with POCL; AIs were a gating issue to the financial success of Pathway.”
I think you have spoken to those already, haven’t you?
Charles Cipione: Yes.
Mr Beer: On design and implementation you say:
“Hardware issues were prevalent during national roll-out. Many post offices were disconnected for extended periods of time; the persistence of reference data management (sic) degraded the integrity of the Horizon system.”
Collecting those three things together, how serious a concern were they?
Charles Cipione: I think that they impacted the actual performance in the real world of the system. We saw in the examples that we just went through, you know, some of it was reference data information that were creating the situation where there were some receipts and payments mis-balance information.
At a more understandable layman’s term level, if your hardware’s not working, if you have just delivered a new system to me and even if the software is perfect but the hardware’s not working, that doesn’t matter to me. I really want is something that works. I want something that works correctly, reliably, consistently and, from an end user’s point of view, I don’t know that it a matters if it was a hardware issue or a software issue or a service provider issue. If the system in my perception is not working correctly, it’s not working.
It doesn’t really matter why it’s not working, and ICL Pathway, you know, at this point is in charge of delivering of that system. So regardless of whether they’re personally – you know, if it’s within their particular enterprise responsibility, or if it’s more of an umbrella partnership with other enterprises, they are going to be the one that get the focus if it doesn’t work.
Mr Beer: “User support. Postmaster training experienced difficulties during national roll-out; Helpdesk was often the root of SLA issues with POCL.”
You have explained that already I think.
“Error resolution: The SMC was frequently cited for not properly filtering calls to the SSC. The SSC was overwhelmed with escalated issues … but were earnest in their efforts to perform their duties.”
You have explained all that.
“Accounting integrity: the persistence of reference data mismanagement degraded the integrity of the Horizon System.”
Was that a view that you formed as being applicable even at the end of the period that you looked at, i.e. as at after roll-out?
Charles Cipione: Yes.
Mr Beer: “A persisting issue related to AI376 (accounting integrity); payment and receipt imbalances were common symptoms with varied causes.”
We have just examined those in section 18 of your report.
You came to those conclusions without, I think, seeing any or all of the reports created at the time by developers, programmers, coders, engineers, within either ICL Pathway or POCL, and went instead off the PinICLs, PEAKs and KELs, and the monthly reports and other management reports.
I think since then you have seen, through looking at the witness statements and reviewing some of there was transcripts of the evidence, a range of other reports or documents referred to; is that right?
Charles Cipione: That is correct.
Mr Beer: Amongst that material that you have reviewed, was there anything you would, in particular, wish to highlight as being relevant to as undermining or supporting the conclusions that you have set out in those six sub-paragraphs?
Charles Cipione: There is nothing that I read that does anything but support my confidence in the conclusions that I’ve written in my report.
Mr Beer: Amongst the material that you examined, did anything stand out in particular to you?
Charles Cipione: I will say that there was a lot of discussion in many of the witness statements and the transcripts that I read that was talking about the design and development process that went on at Pathway for the Horizon programme. Much of it was talking about what appeared to be a lack of design process and more of a “Let’s just go develop it”, without necessarily having a good set of requirements or a proper design facility to help inform that development process.
Now, that’s what I’ve read. I will say that that certainly could substantiate the amount of PinICLs and PEAKs, the volume of PinICLs and PEAKs that were there. It certainly could substantiate the fact that there are – when I was looking at the payments and receipts imbalance and noticing that there were a lot of different causes for that, if in fact there was not really a good design – and I have no reason to not believe what I read in the witness statements – that certainly would explain why there were so many errors in the system as represented by the PEAKs and PinICLs, and why, even when you fixed one, something else popped up, and then you fixed one and something else popped up. That certainly is a consequence of not having an extremely disciplined design-to-development process.
Mr Beer: We’ve heard evidence, in particular, concerning what was called the reverse engineering or reverse documentation of the part of the EPOSS system. Have you read about that evidence?
Charles Cipione: Yes.
Mr Beer: Sorry, have you read that evidence?
Charles Cipione: Yes.
Mr Beer: Did you read the evidence concerning the EPOSS PinICL Task Force?
Charles Cipione: Yes.
Mr Beer: Did you read the EPOSS PinICL Task Force report?
Charles Cipione: Yes.
Mr Beer: I wonder whether we could look at that then, please. That’s FUJ00080690. Is that the same document that we are both talking about?
Charles Cipione: Yes.
Mr Beer: Rather than take you through individual parts of it at the moment, I am going to ask you to look at a couple of bits in a moment, what if any was your reaction to reading it?
Charles Cipione: It was disturbing to read that. If everything in here is as represented, it would indicate to me that, number 1, there was no design at least for the EPOSS section of the Horizon System and, considering that’s what a lot of the PinICLs and PEAKs errors were that we just went through that specifically led to the imbalance issues, it’s – that’s fait accompli. If you’re not going to design something properly, it’s not going to – if you don’t have a good design, it’s not going to work properly.
So, if there’s no design and that design – first of all, there has to be a good design process. Second, that design process has to be aligned with a very well known set of requirements from the sponsors. Only when those two things are done can you start thinking about the construction and development of the software. Otherwise, it’s very difficult to make sure that the system does what your customers want it to do.
Mr Beer: Can we look at a couple of aspects please, page 6 of the document. Scroll down, please, to EPOSS Documentation.
“The document suite supporting EPOSS consists of three main elements:
“EPOSS functional specification.
“High-level design …
“Several low-level design documents.”
And the authors wrote that:
“All of these were developed by reverse engineering the EPOSS product code at that time.”
We’ve been told that that means, not that the code was reverse engineered, but the documentation was reverse documented to fit the code that had already been written. Is that what you were just speaking about as being ill-advised?
Charles Cipione: Yes, because what that implies is you didn’t know how to construct the system from the beginning to make sure that it aligned with the customer specifications. If you need to do this, that means there was not the proper design process for a system of this ambition, because you can’t – this is not like sending three people into a room and asking them to code a system. This is many teams, many teams of coders, and then many other teams, you know, for infrastructure and for telecommunications and for training and for all of that.
If you are doing that by basically putting people in a room and coming in and saying, “I think this is what they want. Why don’t you code for three months and show me what you have.” It’s just … it’s undisciplined.
Mr Beer: Can we go forward to page 15, please. Paragraph 7.1.1 the authors record:
“The EPOSS product was originally developed using RAD techniques.”
You told us about RAD on the last occasion. When you wrote your report, did you know whether or not the EPOSS product had been developed using RAD?
Charles Cipione: No.
Mr Beer: Do you have any view or comment on the appropriateness of using RAD techniques to develop which EPOSS product?
Charles Cipione: RAD is very good for developing something small, for developing something quickly that is just for illustrative purposes. But for a heavy-duty enterprise system that needs to be viable over the long-term, it’s terrible, because you’re not planning it. RAD almost means no plan other than, I think I know what the goal is, and I’m going to get to that goal as fast as I possibly can, and there’s good situations where that’s needed. But this is not the right situation for that.
You need to do everything in a very methodical almost militaristic way to make sure that everything works properly together, because I think I mentioned before: there are so many moving pieces. This has to be highly co-ordinated. RAD is the other end of the spectrum. It’s not highly co-ordinated or it doesn’t support co-ordination amongst big teams because that’s not the purpose of RAD. RAD is: get it done quick, get a little bit done quick. If you have 80 teams doing their own little part independently, you can’t put that back together and think that it will work as well as if they were all very co-ordinated from the beginning.
Mr Beer: Thank you.
Then lastly on this document page 17, please, foot of the page, from paragraph 7.3 onwards there are four examples of some code that the authors have cut into the document. I think, when you prepared your report, you were not asked to look at any of the underlying code, if it’s still available for Legacy Horizon, were you?
Charles Cipione: That’s correct.
Mr Beer: Here you see some example code that the authors have included in their report. Did you read this section of the report?
Charles Cipione: I did.
Mr Beer: You read the code?
Charles Cipione: I did.
Mr Beer: What did you make of it?
Charles Cipione: This is terrible code. This is terrible code. So even the one you have up here – so it does what it’s supposed to do, but it’s like, you may think building – I can’t remember the term – the overly engineered mousetrap. This has to be a joke. I mean, this has to be a joke, because this is a ridiculous set of code to change the sign on something. I’m sure that, even if you don’t code, you would understand that, if I want to make something a negative, I just times it by negative 1. You don’t need an “if” statement. You don’t need to do anything but that. This is, you know 1, 2, 3, 4, 5, 6 – six lines in what could be one line.
So it’s inefficient, at the very least it’s inefficient or it’s actually a joke, because it’s terrible.
Mr Beer: Then very shortly then, if we go over to page 19, two more examples of code there. An example of some unreachable code and what’s described by the authors as bad practice. Did you form a view on either of these two codes?
Charles Cipione: Yes. So the second one is bad practice. It’s just not the right structure, and it indicates to me that they don’t understand what those particular structures are, and just take my word for it. They don’t understand what the structures are.
The first one, I think, is probably a little easier for everyone to understand why this is bad, if you actually can scroll back up, please. So the first one, the highlighted items, when I read this and looked at it, essentially what’s happening here is they are saying, “if” – what they are showing here is, first, they are checking to see if a particular variable was either 3013 or 3016. So that has to evaluate to true.
So, in order to get into that code, you already know that “lstockrootnode” is either 3013 or 3016. Further in the codes then they are checking that same stock root node to see if it’s 2493. It’s not going to have changed – let me make sure.
There’s nothing in here that has changed the value of that particular variable. So there’s no way that it will ever get inside this particular condition.
So I don’t know if this is an artifact of something old where perhaps the first condition was changed and now it’s changed it again. But, even if that was the case, there’s no comment on this. I don’t know what they are checking for in this first condition. So either this is written by someone not so smart in here, or there’s been multiple updates to this code, and perhaps the first statement used to be: if it’s 3013 or if it’s 2493, because I could see that as being a possibility.
But then further on someone went and changed the condition. But then they didn’t account for what happens with 2493. So now am I ignoring what should have happened correctly to 2493, or am I just completely divorced from mathematical logic and put this in here? Either way it’s a bad example.
Mr Beer: And the third example at the foot of the page.
Charles Cipione: So this is a little bit more –
Mr Beer: Niche.
Charles Cipione: – art, art-ish. It just has to do with the construction of, am I going to do – why would I put an “if” statement in a “do” statement? I’m not sure that this would resonate so well with anyone that’s not a programmer, but this is a poor construction.
Mr Beer: Over the page, please.
Charles Cipione: Okay. So, if we remember my first testimony, I was talking about hard coding information into code, and then let’s instead use a data-driven logic. In this code, this is an example of some hard coding, and what this is, this has a whole bunch of values in here that have been hard coded. Why have they been hard coded that way? I have no idea because I don’t see any comments around here.
I believe that the author of this document speculated that there was a particular, very specific condition that was trying to be resolved by doing this hard coding, which means essentially they are looking for any one of these numbers, and then they are doing something, if they find any one of those numbers.
Why are they looking for those numbers? I have no idea. I don’t think anyone that would come after them, after this particular author was there, would have any idea why those numbers are hard coded in there. Can I take them out? What will happen if I take them out? No-one knows.
It’s just very odd that you wouldn’t put this into a variable to at least allow some flexibility. First of all, there’s no flexibility here at all. It’s just these numbers other than, if I wanted to start typing some more numbers into the case statement, but more importantly that’s really it. It’s not very often that you see hard coding in somewhere unless they are trying to quickly fix an issue.
Mr Beer: Thank you. That can come down off the screen. Then my last question just before the break, please, can we go back your report at page 5. And your overall conclusion at 1.1.9, you say:
“In my opinion, the stability of the Horizon IT system was affected by these shortcomings. The sometimes conflicting expectations between ICL Pathway and POCL introduced a disruptive element at the management level. The effects of these disruptions manifested itself throughout the implementation of the Horizon IT System. Other ICL Pathway self-inflicted wounds included the suboptimal support from ICL Pathway’s program for training of SPMs, the Helpdesk support of SPMs and the Helpdesk of ICL Pathway’s error resolution function. A noticeable symptom of these issues was a recurrent balancing problem experienced by the SPMs which directly degraded the accounting integrity of the Horizon IT System.”
Given everything that you’ve read since you made that statement, is there anything that you wish to say to change or to qualify it?
Charles Cipione: No, no. Everything I’ve read has been – no, I’m confident in this statement.
Mr Beer: Thank you very much. Might that be an appropriate moment for the break? I wonder whether we could take ten minutes, in which case I think there will be a good chance of completing Mr Cipione’s evidence today.
Sir Wyn Williams: Certainly. I certainly have no wish to bring Mr Cipione back for a very short period of time tomorrow, so I will sit as reasonably long as is necessary to finish him.
Mr Beer: I don’t think that’s going to be necessary, sir, but thank you anyway.
(3.34 pm)
(A short break)
(3.45 pm)
Mr Beer: Sir, can you see and hear us?
Sir Wyn Williams: Yes, I can.
Mr Beer: Mr Stein first.
Questioned by Mr Stein
Mr Stein: Mr Cipione, I ask questions on behalf of a large group of ex-subpostmasters and mistresses and managers, a group of 156. Mr Cipione, you have given evidence over now some days, your time when you came to the Inquiry before and obviously today. Virtually all the questions I was going to ask you have now been asked. So I can really cut this down.
You were asked number of questions today by Mr Beer about the fact that the system should have been designed to be able to support court proceedings. Do you remember the questions I think earlier this morning?
Charles Cipione: Yes.
Mr Stein: The way that my learned friend Mr Beer put that was that he was talking about cases that are being prosecuted; in other words, prosecuted in the criminal courts.
Charles Cipione: Yes.
Mr Stein: Your evidence was that you’re familiar with such structures because, in fact, you’re engaged in the design of those types of systems yourself; is that right?
Charles Cipione: Yes.
Mr Stein: Now, just so that we can round this particular question out, you’re aware, obviously, that there are both criminal courts and civil courts.
Charles Cipione: Yes.
Mr Stein: The criminal courts work to what the lawyers call the higher standard of proof, which is beyond reasonable doubt, and the civil courts work to a different standard of proof, which is, in the UK, on the balance of probabilities. I think in the States it’s the preponderance of evidence; is that right?
Charles Cipione: Yes.
Mr Stein: Okay. Now, so that we make sure we’re all understanding your evidence about this, when you’re designing a system that should be capable of supporting cases in the courts, essentially do you agree it should be designed to fulfil a high quality product, making sure that material or evidence from it comes from a system which is capable of holding high data integrity?
Charles Cipione: Yes.
Mr Stein: Now, a number of different issues have been highlighted with you as to the problems that were existing within the Horizon System. Now, in the last part of your evidence you were discussing the coding and the difficulties that you were seeing in relation to particular examples. You will be pleased to hear – probably everyone will – I am not going to try and go near those.
Let’s just think about other issues other than software. So let’s start with hardware. So hardware issues – is this correct – such as connection issues or de-connection issues, going offline, power cuts, environmental issues that relate to the weather, all of those issues can also affect the operation of the system in different ways; is that correct?
Charles Cipione: Yes.
Mr Stein: So using just one example, if there’s a power outage, an unexpected power outage, that could cause issues, if there is mis-communication within the system, with balancing; is that right?
Charles Cipione: Certainly it’s a possibility.
Mr Stein: There are also issues that you mention within your report about training. Now, do you accept that this goes in perhaps two different ways: First of all, if you get your training wrong, you’re training people badly, then people, users of the system, in this case subpostmasters, may themselves get it wrong when they are using it; is that right?
Charles Cipione: They could operate the system wrong; that’s correct.
Mr Stein: Then there’s the other side of it, which is that the training itself has not been sufficient, or incomplete, or not sufficient for the people that are operating the system so they’ve not been able to get it or understand it or use the system adequately, yes?
Charles Cipione: Yes.
Mr Stein: So that’s another level of issues that come up within the system, all of that entirely innocently from the perspective of subpostmasters; do you agree?
Charles Cipione: Yes.
Mr Stein: Now, one final thing regarding Helpdesk issues. We’ve seen some evidence – and I can take you to it if we need to – that there were scripts which were inaccurate; in other words, scripts being provided to the Helpdesk advisers that were inaccurate, in other words, that the Helpdesk people were using scripts that were inaccurate and then giving inaccurate advice; okay?
Charles Cipione: So you are saying that Helpdesk scripts were inaccurate or the help was not correct?
Mr Stein: Right. So that can lead to yet another level of issues, because the subpostmaster may accept such inaccurate advice and operate the system badly based on that advice; do you agree?
Charles Cipione: Certainly.
Mr Stein: It also will lead to confusion no doubt within the subpostmasters and their ability to operate the system more generally?
Charles Cipione: Yes.
Mr Stein: The final part of that is: it will probably lead for more contacts back to the Helpdesk?
Charles Cipione: Yes.
Mr Stein: Now, you mentioned that we are looking at a system that was designed and started in operation at the end of the last millennium. That was part of your evidence today.
If the system was designed so that a customer going into a post office branch was able to have their transaction completed and verified back to the Horizon, if you like, mainframe, back to the main servers, and then confirmed to the subpostmaster so that all of that was done in one basic transaction, so confirmed from both ends that the information is correct, that there has been receipt of that information at the branch, confirmed back to the office, if you like, and confirmed yet again back within that same transaction to the subpostmaster, would that be the only way that you could design a system at that time, so that everyone knew everyone’s checking going, “Yes, you got it right, it’s all clear”?
Charles Cipione: I think your goal of the confirmation is a good goal. How you would effectuate that in the late ’90s, I think, would require some thought on my process, but I appreciate what your goal is, and I think that would be a good goal.
Mr Stein: Was the system designed to do that?
Charles Cipione: I really can’t comment on exactly what the mechanics of the system are. I could say that I don’t think it was designed to do that, based on my limited understanding of the technical specifications underneath.
Mr Stein: Can I point you in one direction that may help. We know that the system was designed so that, even if there was connectivity lost, the branch could continue to operate, working on the basis that the branch, when reconnected to the main system, would then catch up; in other words, there would then be a conversation.
Charles Cipione: Yes, I’m aware of that functionality.
Mr Stein: So that tells us that, in my example of a customer coming into a branch that, if the system happens to be operating offline at that particular time, there wouldn’t be a capable way of actually communicating confirmation and acceptance, would there?
Charles Cipione: No, no, there would not.
Mr Stein: Now, you’ve been referred by Mr Beer at paragraph 7.4.5 – if we need to go to it, it’s the reference to correspondence that some KELs may have been deleted. You were asked a couple of questions about that.
Just so we understand the detail about this, what you say in your report is that some KELs may have been deleted. That’s your understanding from the correspondence.
Were you given any information about whether they were or were not deleted? The word “may” is rather loose.
Charles Cipione: I think I’d have to refer you to Mr Beer. I didn’t have any direct correspondence with Fujitsu.
Mr Stein: So we need get back to the Inquiry and ask them the detail about that conversation.
Charles Cipione: Yes.
Mr Stein: Lastly, this happens to be at 6.2.20, you discuss the failure by Fujitsu to provide PinICLs in a timely fashion. Can you help any further with that and how that happened?
Charles Cipione: What specifically are you referring to?
Mr Stein: In your report at 6.2.20 – if you can have the report up on the screen, please – so reference EXPG, thank you, and then pagination Relativity pagination page 60. We’re looking for 6.2.20. If you can highlight 6.2.20 just further down. Thank you.
To this is the point I am referring to:
“The PinICL archive databases were received late in my review. I therefore decided, in consultation with the Inquiry team, not to fully investigate the databases as this would have unduly delayed the completion of this Report, and could have had knock-on consequences for the Inquiry’s timetable. In addition, I noted that Fujitsu had not produced these PinICL archive databases in response to the original Rule 9 request submitted by the Inquiry in December 2021. Therefore, I deduce the incremental information not to be responsive to the original request from the Inquiry.”
If we just quickly take this into bits, first of all did you ever receive those PinICLs?
Charles Cipione: Yes, I have those PinICLs and perhaps an explanation might clarify that. So we received the 50,000 or 60,000 PinICLs and PEAKs in PDF or HTML format. Subsequent to that, we were delivered this database which was, for the same PinICLs and PEAKs, just in a structured format. It wasn’t – it wasn’t new PinICLs and PEAKs; it was just in a different format.
Mr Stein: So in the end the confusion about this was cleared up?
Charles Cipione: Yes.
Mr Stein: You were able to look at the PinICLs fully?
Charles Cipione: Yes.
Mr Stein: Thank you, sir.
Questioned by Ms Page
Ms Page: Laura Page, I represent a number of postmasters as well, and just a few questions to see if there are a couple of areas you may be able to help me with; you may not.
One of your 7 examples that you talked through with Mr Beer earlier is dated from June 1999, and it was an experience that’s common to a number of the postmasters. It involved the bringing forward of balances and, when you brought them forward, they doubled up. Then you might bring them forward again and they’d double up again, and that’s something that we have heard a lot of from the subpostmasters.
Your example was from June 1999. One of the subpostmasters that I represent, she had that problem in 2000 – Nichola Arch. So not very long afterwards but nonetheless perhaps a year or so afterwards. Then we see it happening again and again.
In the example that you looked at, it appeared that a fix had been applied, but we see the problem coming back.
Now is that something that we can relate to this terminology that we’ve heard “code decay”, where you fix on fix on fix, and you end up sometimes making the problem worse, or it comes back in a different form? Does that make any sense to you?
Charles Cipione: It’s a possibility. Of course I can’t tell you exactly why, because even in the examples I reviewed, I didn’t see the code that – if that example was code related.
If some of the witness statements and transcripts I’ve read indicate that, you know, regression testing wasn’t done properly. It is possible that the issue that I was describing in the PinICL was corrected and then for some reason someone went back in and did another correction to that code but actually reverted it back; that’s a possibility. I have no idea if it happened but it’s certainly a possibility.
Ms Page: Thank you.
We also heard from a witness, Mr Andrew Simpkins. I don’t know if you read his testimony, but he talked about an element of the design which he saw as a pretty fundamental design flaw in which the problem was that subpostmasters couldn’t see where or when discrepancies occurred in their accounts. It was completely opaque to them. So numbers just kind of came out, but they didn’t know the calculation behind that or where those numbers had been put in.
He said that that would have compounded a problem because, when they called for assistance, they wouldn’t be able to identify exactly when the problem occurred, leaving the Helpdesk unable to – I think the word he said, was that “replication” of the fault is so important to try to get to the bottom of the fault that this may well have compounded the problems that the Helpdesk were having because subpostmasters weren’t able to say, “Well, it happened exactly like this”.
Is that something that you would agree with, that was a kind of a design flaw, if you like?
Charles Cipione: I would say that that was a usability design flaw certainly. I don’t know that that is necessarily a technical design flaw. I mean, I’m kind of picking at nits here but, from a user perspective, absolutely. If I could have seen a register of everything that the system was recording that I had entered, even if I hadn’t entered it, it certainly couldn’t have hurt the investigation process. It absolutely most likely probably would have helped the investigation process.
Ms Page: Thank you. Just a couple more. Again, I hope this isn’t a lengthy one but it may assist if we bring up a document, please. The reference is POL00043744.
Sir Wyn Williams: Ms Page, could I ask you to speak a little more into the microphone, please.
Ms Page: Certainly. Is that better?
Sir Wyn Williams: Yes, it is. Thanks.
Ms Page: So the document that I’ve just brought up if we go to page 30 and we go down, please, towards the bottom of the page and going over to the next page, we see a section that is headed “audit and Fraud Management” and this deals with – this is a document dated in 2001 and it deals with the fact that when the Benefits Agency came out of the tripartite agreement, one of the things that happened was that a fraud management system that they had wanted and had sort of put in for benefits fraud was removed.
If we could just perhaps read about one of the possible implications of that, it says:
“There are a number of possible implications on the Horizon audit solution.
“Is there a need for the Audit Server to harvest and store information from the NBE? If so, then the NBE will need to support either NFS or NT file shares.
“Audit Server sizing will need to be examined. There will be a significant increase in the data that needs to be audited.”
If we could go over, please:
“Will there be an increase in the number of requests to retrieve audit information? The current Audit solution is designed to service a limited number of requests (a maximum of 50 per year). Any dramatic increase in requests will require the current solution design to be re-examined.”
Then:
“A fraud management system was developed for the PAS/BES system, but removed from the Horizon solution when these applications were dropped. It is necessary to consider whether there will be a requirement for fraud investigations and control with Network Banking. Although the current Audit solution will hold the data that may form the basis for detailed fraud investigations, it is not designed for, nor can it cope with, an ongoing fraud management approach based upon statistical analysis of transaction patterns over a wide variety of Outlets.”
Now, I don’t pretend to understand all of that and I don’t know whether you would be able to help us with all of it either, but what it appears to be getting at, as far as I can tell, is that with the removal of the fraud management system, there’s then set up a sort of an arrangement we’ve heard about where up to around 50 requests a year for audit were done in a sort of a much less organised fashion, and those 50-odd requests could be used by Post Office for all sorts of things. It might be just an investigation, it might be what’s been told, a fishing expedition, or it might involve actually going forward to a criminal prosecution.
What I am trying to understand from this set of paragraphs is, is there something about the maximum of 50 a year? Is there something magic about that? Does data degrade in some fashion if there’s going to be more than 50 requests a year?
Charles Cipione: Who was the author of this document?
Ms Page: This appears to be an ICL Pathway document trying to deal with the Network Banking, the next phase. So it’s a later phase and they’re talking about moving forward into that phase but one of the things they are dealing with is this issue around how do we support prosecutions and investigations, how do we provide audit trails?
Charles Cipione: Okay. So, I’m sorry, what was your question again?
Ms Page: So I’m trying to understand whether this 50 – in the bullet point that’s still on the screen:
“The current audit solution is designed to service a limited number of requests (a maximum of 50 per year) any dramatic increase in requests will require a current solution design to be re-examined”, can you sort of see any reason why there was – is there going to be a problem with extracting data if you don’t have a proper solution for it and is there some magic around the number of 50?
Charles Cipione: First, I don’t even know what this system is, so I don’t know that any of my commentary is helpful or not.
Ms Page: No.
Charles Cipione: I don’t know why that 50 is in there but I also don’t know what the system is, so anything I say would just be speculation.
Ms Page: All right. Then just finally this also might be a bit speculative, but it’s just in case you are able to assist.
On the non-polling issue, when I put that section of your report to a witness who was a network witness (that was his specialty), Mark Jarosz – I don’t know, again, if you were able to hear much or read much of his – but he explained that when non-polling occurred there was a procedure which ought to have been in place which ought to have involved a special laptop being brought to the branch in question and he said that there was a solution, a Day D solution.
I should say that the subpostmasters we’ve heard from, I don’t believe any of them have experienced this occurring but is there something that – does that ring any bells? Is that something that would have worked if a special laptop has arrived that would have helped with the data reconciliation?
Charles Cipione: I’m not familiar with that. I suppose they could have brought another piece of hardware and connected it up, but I’m not familiar with what you’re discussing.
Ms Page: All right. Thank you. Those are my questions.
Mr Beer: Sir, I believe that’s all of the questions for Mr Cipione. Can I thank him on behalf of the Inquiry for his report and the two occasions when he’s come to this country to give evidence. Thank you.
Sir Wyn Williams: Well, you can do it, Mr Beer, but I think I’m not just duty-bound but it’s my pleasure to give Mr Cipione my considerable thanks, and I break it down in this way, Mr Cipione. First of all, for the production of a comprehensive report in a very tight timescale, so my thanks for that, and I dare say thanks are due not just to you personally but to the team that you had helping you in that regard.
Then, secondly, I’m very grateful for you keeping your eye on the Inquiry and reading evidence as it’s unfolded so that you could verify or alter or amend in any way you thought appropriate evidence contained within your report; and then, lastly, for coming twice to answer a great many questions for which you, again, have my considerable thanks. Thank you very much.
Charles Cipione: Thank you, sir.
Mr Beer: Sir, I think we return next Wednesday at 11.00 am.
Sir Wyn Williams: That is correct. Thank you very much, Mr Beer.
Mr Beer: Thank you.
(4.08 pm)
(Adjourned until 11.00 am on Wednesday, 23 November 2022)