18 October 2022 – Charles Cipione
Hide video Show video
(10.00 am)
Mr Beer: Good morning, sir. Can you see and hear me?
Sir Wyn Williams: Yes, I can. Can you hear me?
Mr Beer: Yes. Thank you very much. Can I call Charles Cipione, please.
Sir Wyn Williams: Yes.
Charles Cipione
CHARLES CIPIONE (sworn).
Examined by Mr Beer
Mr Beer: Thank you very much, Mr Cipione. Please take a seat. Can you give us your full name, please?
Charles Cipione: Charles Anthony Cipione.
Mr Beer: You’ve been instructed as an expert witness by this Inquiry and you’ve kindly prepared a written report as a result of those instructions; is that right?
Charles Cipione: Yes.
Mr Beer: Please can we see your report of 14 September 2022. Thank you very much. For the record, it’s EXPG0000001. You’ve got a hard copy in front of you, in case you need it, but it should be displayed on the screen as well. Is that the first page of your report?
Charles Cipione: Yes.
Mr Beer: Including appendices, is it 174 pages in length?
Charles Cipione: Yes.
Mr Beer: I think it’s divided into two parts, Parts 1 and 2, and Part 1 is between pages 12 and 64, if you could just turn that up and confirm, please?
Charles Cipione: Yes.
Mr Beer: So that’s Part 1 beginning on page 12 and then, if we go forward to 64, we can see the end of Part 1, and then Part 2 is between pages 65 and 160 –
Charles Cipione: Yes.
Mr Beer: – when the appendices begin on page 161 and I think there are five appendices, A, B, C1, C2, and C3?
Charles Cipione: Yes.
Mr Beer: As you know, you’re kindly giving evidence to us, Mr Cipione, in two stages, today and possibly tomorrow, about the matters addressed in Part 1 of your report, and then on 17 and 18 November this year you will return to give evidence about Part 2 of your report; do you understand that?
Charles Cipione: Yes.
Mr Beer: Can you confirm the following, please: firstly, you’ve made clear in Part A (sic) of your report, which facts and matters referred to are within your own knowledge and which are not.
Charles Cipione: Yes.
Mr Beer: Secondly, that those facts in Part 1 of your report that are within your own knowledge, you confirm to be true?
Charles Cipione: Yes.
Mr Beer: Thirdly, that the opinions you’ve expressed in Part 1 of your report represent your true and complete professional opinions on the matters to which they refer?
Charles Cipione: Yes.
Mr Beer: Thank you. Can I begin by asking you your qualifications and experience as an expert witness. I think you hold a Bachelor of Science degree from Texas A&M; is that right?
Charles Cipione: That is correct.
Mr Beer: The A&M standing, I think, for “agricultural and mechanical”?
Charles Cipione: That is correct.
Mr Beer: You hold a Masters of Business Administration, an MBA, from the same university?
Charles Cipione: That is correct.
Mr Beer: I think your professional career began at Arthur Andersen, a firm headquartered in Chicago and which, at its height, was one of the largest public accounting firms in the 1990s with 85,000 employees operating across the world; is that right?
Charles Cipione: Yes.
Mr Beer: Is it right that you worked in the information systems risk management business within the firm?
Charles Cipione: Yes, that is correct.
Mr Beer: What did your work then involve?
Charles Cipione: The work that we performed while I was at Arthur Andersen was around systems controls, a lot of general controls reviews which simply means making sure that the proper policies and procedures surrounding the financial systems at firms were in place and allowed for the auditors to rely upon the information that was in those systems. Also performed a number of application controls tests, which was a little bit more in-depth, on individual applications, whether they were efficacious or not, and various – it really depended on which project I was working on but it was generally to look at the efficacy of systems.
Mr Beer: You say in your report you developed and implemented database applications and analyses relating to litigation and bankruptcy clients?
Charles Cipione: That is correct. In addition to the audit projects that I’ve just mentioned, I also developed – also worked a lot with our litigation and our bankruptcy group in developing, maintaining – developing, deploying and maintaining database applications related to either companies that we had been hired to help through the bankruptcy process in the United States or companies that hired us to perform expert work in litigation – in the litigation arena.
Mr Beer: I think, in due course, you left Arthur Andersen and set up your own consulting firm, Cipione & Associates?
Charles Cipione: That is correct.
Mr Beer: When was that?
Charles Cipione: That would have been around 1994.
Mr Beer: What did that venture involve?
Charles Cipione: That was – basically I was a software developer. I was – for various clients, I would design, develop, deploy and maintain software applications.
Mr Beer: You mentioned in your report, for the notice 2.1.4, that the software you designed, developed and maintained was DOS. What is or was DOS?
Charles Cipione: DOS stands for Disk Operating System. It’s basically the operating system for PC-based computers that was developed by Microsoft.
Mr Beer: It runs from a disk drive; is that right?
Charles Cipione: Yes, that’s correct.
Mr Beer: This was a Microsoft product. The predecessor to Microsoft Windows; is that right?
Charles Cipione: That’s correct.
Mr Beer: I think in 2001, you joined AlixPartners; is that right?
Charles Cipione: That’s correct.
Mr Beer: Can you explain, please, who or what AlixPartners is?
Charles Cipione: AlixPartners is a global consultancy that performs a variety of services for clients, probably at that point in time was best known for their turnaround or restructuring services within the United States. They basically would take over companies that offered Chapter 11 protection within the United States and operate those companies through the bankruptcy process, which often terminated in a plan of confirmation to get those companies out of bankruptcy.
Mr Beer: Thank you. You say that when you joined you helped establish the claims management service; what did that involve?
Charles Cipione: Yes, so a big part of the bankruptcy process in the United States has to do with the reporting of all of the assets and liabilities of the debtor company that’s going into bankruptcy, for purposes of allowing the creditors to understand the debtor’s position on amounts that they think that they owe the creditors.
There are a number of – there are several reports that are required by the court, AlixPartners did not have a system to do that in the group that I belonged to. As we were acquired by AlixPartners, I took over all of the administrative responsibilities for reporting in the court.
Mr Beer: Thank you. You say this involved interrogating, collecting and organising vast amounts of disparate, financial and operational data from your client systems; is that right?
Charles Cipione: That is correct.
Mr Beer: For what kind of clients was that service established and operated?
Charles Cipione: So that’s the bankruptcy process that I was just referring to. Some example clients that we worked on were WorldCom, General Motors, Kmart, there are a vast number of them, but all very large, very major bankruptcies from the 2001 to current time frame.
Mr Beer: Were you the architect of those systems?
Charles Cipione: Yes.
Mr Beer: Are they still in use today?
Charles Cipione: They are.
Mr Beer: I think you’re presently a managing director within the risk analytics group at AlixPartners; is that right?
Charles Cipione: That’s correct.
Mr Beer: You’ve held that position for over 15 years?
Charles Cipione: Yes.
Mr Beer: Is it right you’ve been retained in that position by clients to provide factual and expert evidence in relation to the efficacy of application systems and the management and analysis of datasets relating to litigation and regulatory issues?
Charles Cipione: Yes.
Mr Beer: I think, although it’s right that you’ve plainly given expert evidence before, it’s fair to say that you are primarily a practitioner rather than somebody who spends most of their time in the courts; is that right?
Charles Cipione: That is correct.
Mr Beer: Overall, therefore, you have some 30 years’ experience in information technology; is that right?
Charles Cipione: Yes.
Mr Beer: Can I turn secondly to look at your instructions. You’ve been given two sets of instructions by the Inquiry legal team. The first of them provided to you – I’m at paragraph 2.3 – on 2 June 2022 and then addendum instructions on 27 July 2022; is that right?
Charles Cipione: Yes.
Mr Beer: Are those instructions fairly summarised – can we display this, please – paragraph 2.3.3 of your report, which is at page 7?
Towards the foot of the page, 2.3.3. Does that, in paragraph 2.3.3, in (a), (b) and (c), fairly summarise your instructions –
Charles Cipione: Yes, it does.
Mr Beer: – namely an introduction to the Horizon System and other key terms that will seek to assist this Inquiry in understanding the substance of your report and other submissions that might be made to the Inquiry. You were instructed that the introduction to the Horizon System should be tailored so as to be understandable to the Inquiry, the Core Participants to the Inquiry, to members of the public who may not have prior knowledge of the Horizon IT System; is that correct?
Charles Cipione: Yes, that is correct.
Mr Beer: Is that essentially Part 1 of your report?
Charles Cipione: That is part 1, yes.
Mr Beer: You were instructed to analyse and identify – sorry, and illustrate any themes in the problems that were being experienced by users, in the period up to and including the rollout of the Horizon IT System, including how those problems were resolved or escalated, and the key individuals who were involved in these processes; is that essentially Part 2 of your report –
Charles Cipione: Yes.
Mr Beer: – taken together with the third thing: any overall observations or conclusions, that were within your professional expertise, as to the themes that you identified and the potential reasons for them?
Charles Cipione: Yes.
Mr Beer: You say in your report that, although those were your instructions and therefore provided the basis for the determination of the scope of your work, you’ve nonetheless been responsible as an independent expert for developing your own approach to the questions posed by the instructions; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: Did you undertake your review of the material between June and September this year –
Charles Cipione: Yes.
Mr Beer: – assisted by a team from AlixPartners, including colleagues in the United Kingdom?
Charles Cipione: Yes.
Mr Beer: They’ve assisted you, I think, with anglicising some of the phrases, alternative phrases in the report?
Charles Cipione: Indeed. They did.
Mr Beer: Thank you, so spelling words like “colour” –
Charles Cipione: Yes.
Mr Beer: – and “defence” and things like that, presumably?
Charles Cipione: Yes.
Mr Beer: Okay. In terms of the materials relied on, are they listed over pages 161 to 165 of your report? That’s appendix H, 161-165.
If we could turn those up, please. Thank you.
Charles Cipione: Yes.
Mr Beer: We can see on page 161 a list of PinICLs or PEAKs, and then over the page, please, to 162 we can see the list of PinICLs and PEAKs continued; is that right?
Charles Cipione: That is correct.
Mr Beer: For now, I think, it’s sufficient to know that a PinICL was a customised incident logging and resolution tracking system initially adopted by Fujitsu between I think ‘96 and 2003?
Charles Cipione: That’s correct.
Mr Beer: We’ll come back to these in detail later. Then the PEAK was the customised incident logging system designed to replace PinICL in, I think, 2003?
Charles Cipione: Yes.
Mr Beer: These PinICLs and PEAKs were a selection taken, I think, from some 55,000 such documents that you were provided with?
Charles Cipione: Yes.
Mr Beer: I think, as you tell us in the report, you used computer assisted technology to review that material; is that right?
Charles Cipione: That is correct.
Mr Beer: Because what you describe as a “brute force approach” wasn’t possible and was inadvisable with that volume of data. By “brute force”, in this context, do you mean reading and analysing every one of 55,000 error logs?
Charles Cipione: That is exactly what I mean.
Mr Beer: Rather than the algorithm Brute Force?
Charles Cipione: That’s right.
Mr Beer: Okay, got it. Then on 162 you continue by listing some monthly reports from Pathway and ICL Pathway?
Charles Cipione: Yes.
Mr Beer: Then if we go to 163, please. We see those reports continue, and then a list of other background materials that you had regard to. Then over the page to 164 and 165, some publicly available materials that you list?
Charles Cipione: Yes.
Mr Beer: Your work and therefore the observations and conclusions in the report, and the evidence you’ll give today, are based only on the documentary evidence and data provided to you by the Inquiry, which in turn was provided by some of the Core Participants; is that right?
Charles Cipione: Yes.
Mr Beer: That was primarily Fujitsu; is that right?
Charles Cipione: That is correct.
Mr Beer: Primarily in the period from July ‘96 to December 2000?
Charles Cipione: Yes.
Mr Beer: Thank you. Can I turn thirdly to the scope of Part 1 of your report and the evidence that’s to be given today. Your instructions relate specifically to Phase 2 of the Inquiry, and this is paragraph 2.4.2 of your report, and therefore address the procurement, design, pilot, rollout and modification of and to the system?
Charles Cipione: Yes.
Mr Beer: Like this Inquiry, you’ve adopted the umbrella term “Horizon System” and “Horizon IT System” that was employed by Mr Justice Fraser in his Horizon Issues judgment –
Charles Cipione: Yes.
Mr Beer: – which is:
“… the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branches were processed.”
Is that right?
Charles Cipione: That is correct.
Mr Beer: Now, I think Part 1 of your report is itself divided into four parts. In section 3 of your report, or the paragraphs beginning with 3, you address the theory of system design and development?
Charles Cipione: That is correct.
Mr Beer: Just tell us why is that important, the theory of system design and development?
Charles Cipione: Understanding, especially for people who are not familiar with the intricacies of system design development, deployment and maintenance, it’s important to have just a general overview of what goes into that and what to expect from that process. So I felt it was important to just spend a little bit of time in my report to explain some concepts that I feel will be salient further on in the report.
Mr Beer: Thank you. To be clear, that’s a theoretical or ideal situation that you set out, ie the paradigms of design, et cetera, rather than relating to this system?
Charles Cipione: That’s right. This is all theory. It has nothing to do with the actual documents I reviewed.
Mr Beer: In section 4 of your report you introduce the Horizon System. You explain in summary terms what the system is, how it was structured and how the system evolved over time?
Charles Cipione: Yes.
Mr Beer: Just by way of summary, on page 9 of your report, the paragraph at the top in the (a), (b) and (c) – thank you – you detail in summary form the three major iterations of the Horizon System; is this right?
Charles Cipione: That is correct.
Mr Beer: Firstly, the original system introduced into branches from 1999 onwards and active until 2010, now known as Legacy Horizon, although presumably not known as Legacy Horizon at that time?
Charles Cipione: That’s correct.
Mr Beer: Then the first major iteration of the Horizon Online system, known as HNG-X, which was introduced in 2010 and active until around 2017?
Charles Cipione: Yes.
Mr Beer: Then the second major iteration of the Horizon Online system, introduced in about 2017 and still active today, HNG-A, which I think is Horizon Anywhere?
Charles Cipione: Yes.
Mr Beer: The third thing you do, in section 5 of your report you introduce Horizon’s error logging and remediation systems?
Charles Cipione: Yes.
Mr Beer: Then in section 6 you explain in more detail the materials provided to you?
Charles Cipione: Yes.
Mr Beer: Thank you.
Can I turn to the limitations on your report. In paragraphs 2.7.1 to 8, which is on pages 10 and 11 of your report, you identify a series of limitations to your report and therefore of the evidence that you can give today and in November; is that right?
Charles Cipione: That is correct.
Mr Beer: I think summarising them, there are, I think, six or seven of them: firstly, the documentation on which you relied was a quarter of a century or so old, was written for an internal market and not for the purposes of subsequent forensic examination in legal proceedings; is that right?
Charles Cipione: That is correct.
Mr Beer: Secondly, the documentation relates to, principally, the period from ‘96 and 2000, reflecting the focus of your report being on the rollout of Legacy Horizon?
Charles Cipione: Yes.
Mr Beer: Thirdly, given the nature, extent and duration over time of the Horizon System, you could have spent an unlimited amount of time researching and analysing it?
Charles Cipione: Indeed. That is correct.
Mr Beer: I think you’re going to tell us in a moment that the documentation related to Horizon – that’s training manuals, operating instructions and the like, the Horizon documentation – itself amounts to over 100,000 documents?
Charles Cipione: That is correct.
Mr Beer: That’s documents not pages?
Charles Cipione: Yes.
Mr Beer: Fourthly, it was in the nature of the task that you undertook that you were focusing on material that tended to describe problems and difficulties, rather than trumpeting the accomplishments of Horizon?
Charles Cipione: Yes.
Mr Beer: Fifthly, is this right, given the technical nature of the error logs, PinICLs, PEAKs and KELs, you may have missed nuances or subtle shades of the use of language within them, which nuances and shades may have been evident to those responsible for actually using the system?
Charles Cipione: Yes, that is correct.
Mr Beer: Sixthly, the PinICLs and PEAKs that you examined came from the third line of Fujitsu’s IT support and, therefore, you didn’t examine records relating to the first and second lines of IT support?
Charles Cipione: That is correct.
Mr Beer: Lastly, as you’ve observed already, most of the material you examined originated from Fujitsu and not the Post Office and so you don’t have any insight into Post Office’s views during the period which you’re examining?
Charles Cipione: That is correct.
Mr Beer: Can we turn, then, to the first part of your report, which is section 3 on page 13.
Can that be displayed, please. Can we highlight 3.1.1, please. Thank you.
You say, and I’m going to read it into the record, that:
“To properly understand software systems, it is important to appreciate how they fit into the overall execution of the enterprise they support. Software systems are enablers, not panaceas. In the best situations, software applications can decisively improve the execution of the enterprise’s strategy by streamlining operations. This often includes providing complete and accurate reporting that informs decision makers in a timely manner. In the worst situations, mismatched expectations and/or faulty designs and implementations degrade the execution of the enterprise.”
Can you explain what you were conveying in that paragraph, please?
Charles Cipione: Certainly. I believe many people think that software cures everything, that software is the leader of the execution of an enterprise. What I’m trying to emphasise here is that software is a tool that the enterprise should be using in order to execute the strategy and tactics that it has pre-defined, rather than the other way around. The software does not define the strategy and tactics; the software is a servant to the strategy and tactics of an enterprise.
Mr Beer: You then set out, over the following paragraphs, the five components that permit execution of the enterprise and the first of those, the model components, is strategy. Can you explain what you mean by “strategy”, please?
Charles Cipione: Strategy, as I say in my report, is the very high level driver of what, you know – what an enterprise is trying to accomplish and the way it’s trying to accomplish it. This is often used, you know – or often encapsulated in mission statements and vision statements, and it is a very – usually a very straightforward, simple to understand set of concepts that is the DNA, basically, of an enterprise and what they’re trying to do and, in general, how they’re trying to accomplish it.
Mr Beer: So this is in the form of a mission statement or statement of purpose. It’s focused on the organisation and not the IT system?
Charles Cipione: That is correct.
Mr Beer: You set out at the foot, it’s on the page now, the foot of this page, the UK Post Office’s Statement of Purpose as at the time that you were writing your report. At I think the top line of it is:
“We’re here, in person, for the people who rely on us.”
It goes on to explain what it means by those three component parts.
Charles Cipione: Yes.
Mr Beer: We needn’t read those, but that is a Post Office strategy?
Charles Cipione: Yes.
Mr Beer: You then explain – over the page, please – the tactics or business operations of an enterprise and you say, in paragraph 3.2.2, I’ll read it in:
“To execute the strategy, it is important to have a mature and well-understood set of policies and procedures. Designing, developing and implementing the tactical playbooks that control the day-to-day business operations across all aspects of the enterprise takes considerable effort. The balance between aspirational goals and realistic constraints is the responsibility of those put in charge of making ‘real-world’ decisions that affect how an enterprise is operated.”
Again, this: the tactics need not refer to the IT system?
Charles Cipione: That’s correct.
Mr Beer: It might do but it does not necessarily do so?
Charles Cipione: That’s right.
Mr Beer: The tactics would obviously be guided by the strategy?
Charles Cipione: Yes.
Mr Beer: You explain, as a third component part, in paragraph 3.2.3, about the concepts of software systems and you say:
“A software system’s sole purpose is to efficiently reinforce the business operations.”
Charles Cipione: That is correct.
Mr Beer: So the tactics select software systems based on their ability to conform to the defined business operations requirements of the tactics and the strategy; is that right?
Charles Cipione: Yes.
Mr Beer: You then speak to the fourth component part, paragraph 3.2.4, “Data Management (Facts)”, what did you mean by “facts”?
Charles Cipione: Oftentimes, information systems will be systems of record. For example, if it’s an accounting transaction, it will record that transaction and I would consider that particular set of data a fact for the enterprise.
Mr Beer: So data management is governed by the design specifications of the software systems?
Charles Cipione: Yes.
Mr Beer: You say in the second sentence there:
“The management of these facts requires alignment of the software systems to the business operations and anticipates downstream analytics and reporting.”
What did you mean by that second sentence, please?
Charles Cipione: So, oftentimes, facts are accumulated in a voluminous nature and the – one of the benefits of having a software system collecting all of this information is to further analyse and report on it. In order to do that correctly, number 1, the software system has to direct the collection of data in a structured and understood manner so that the reporting and analytics that can be performed on that is well defined and well understood by everyone throughout the enterprise.
Mr Beer: You turn, fifthly, to analytics and reporting and this part of the model represents how the enterprise understands the data collected and managed through a series of manipulations and summaries of the data itself; is that right?
Charles Cipione: That is correct.
Mr Beer: Those rely on the rules employed by the data management function?
Charles Cipione: Yes.
Mr Beer: You explain the hierarchical relationship between these components in 3.4.2 of your report on page 15. You say that the two concepts that should be considered that affect a healthy, long-term relationship between the components are adaptability and complexity. Under “Adaptability”, in (a)(i), you say:
“The downstream components should respond to the requirements of the upstream components, not dictate them …”
Can you explain, please, what you mean by “downstream” and “upstream” components?
Charles Cipione: Certainly. So the relationship or the hierarchy within the model that we just went through has a clear pecking order: strategy guides the tactics, tactics selects the software, software controls the data management and the data management supplies information to the reporting and analytics section.
So that’s the relationship that I’m referring to.
Mr Beer: Which of those are downstream and upstream?
Charles Cipione: So strategy is at the top of the hierarchy and reporting and analytics is at the bottom of the hierarchy.
Mr Beer: What do you mean by “not dictate them”? Can you give an example please?
Charles Cipione: So what I mean is there should never be an instance where, let’s say, the reporting and analytics defines what the strategy should be. The reporting and analytics should always be responsive to the strategy, not dictating the strategy. That works the same up and down the hierarchy that I’ve described. The software system should never dictate the tactics of an enterprise. The tactics of an enterprise should always be in charge of the software system and what the software does.
Mr Beer: You actually give an example in (ii) there. Could you flesh that out a little bit, please?
Charles Cipione: Certainly. So in this example what I’m describing is that if the reporting and analytics took it upon themselves to expand on the information that was to be collected, in theory, that should have been guided by the tactics and then were sponsored by the software system. It is possible that there are situations where someone in the reporting and analytics division of this hierarchy felt as though “Well, it would be very nice to have this particular piece of information available but it’s not being collected by the software system, so we’re going to take it upon ourselves to start collecting that information”.
That, in the short run, could be a very good idea. However, as that type of attitude towards the enterprise goes on and on, you find that you’re doing a lot of one-off, ad hoc additions in the wrong place – and in this situation, I mean in the reporting and analytics – of collecting information, and not necessarily everyone within the enterprise even knows that you’re collecting that information and they certainly aren’t governing it. There aren’t any rules governing the collection of that information, at least not from the purview of the strategy or the tactics section.
So, in my experience, what I found is when that happens it’s almost as though a new kingdom has been set up in the wrong area of the hierarchy and, over time, it becomes disjointed with the strategy and tactics of the enterprise and it creates an unstable situation because once the strategy – you know, once the senior leaders or the line leaders of a system are aware that this information is available and start relying on it, but it’s not really being controlled properly through the software system or the data management component of the process.
Oftentimes the integrity of that information is not the best, and the situation then arises that no one knows, really, who is in control of that information. No one has a view onto the efficacy of the information, and it is, essentially, out of control. The process – you’ve introduced a lack of control into the process, and that has a lot of knock-on effects down the line, and any one addition like that seems fairly innocuous but, over time, it’s as though bile is being collected in the system and, eventually, things become very out of control, if you are not adhering to who’s in charge and where the proper division of labour should be for that particular example.
Mr Beer: I think you mention some species of that bile in the last sentence of (ii):
“… inefficiencies of communication, maintenance and costs.”
Charles Cipione: That’s right because, once you start going down that path where you’re out of control, as far as the collection, you know, the proper collection and maintenance of that data, then people are expecting – like, for instance, if this was done completely in the reporting and analysis section, people further upstream, such as in the tactics or in the strategy section might assume that it’s being – that it’s part of the software system or part of the data management system, where it’s not. And the data –
You know, it’s extremely inefficient to have multiple people doing the same thing and what I’m trying to explain here is that, as these lines are blurred, no one knows really who has responsibility over the data, and no one knows where the data is coming from, which is the communications issue, and as systems are upgraded.
So let’s say that we did collect a whole bunch of extra data in the reporting and analysis section and then went through a change management – a change process in the software and the data management section, they have no idea that this other information is being collected and they don’t know how – they wouldn’t know how a change in the software or a change in the policies and procedures that are in the tactics section, or even a change in the data management portion of the hierarchy are affected by the expectation that this extra data is always being collected because they, in fact, might not even know that this data is being collected.
Mr Beer: The second concept to which you refer, to ensure a healthy long-term relationship between the component parts, is complexity. You say in paragraph (b)(i):
“Current efficiency and future flexibility benefit from complexity being localised as far as downstream as possible.”
Can you explain, please, what you meant by that?
Charles Cipione: Certainly. So the systems are complex. You know, running an enterprise is complex. But it’s important for guiding principles at the top to then – as it filters its way down through the hierarchy, to then be more real world. The concept of pushing complexity down as far as possible allows rapid changes to be made to the system, whereas – opposed to if all of the complex existed at the top of the hierarchy, it would require a vast amount of changes downstream because every change at the top cascades down.
So, to the extent that you can push down the complexity as far as possible, it limits the amount of adjustments that need to be done as things are changing, you know, on the different parts of the hierarchy.
Mr Beer: Thank you. Again, in (ii), you give an example of not adopting that philosophy. Can you remind yourself of that example and then try to explain it to us, please?
Charles Cipione: Right. In this example what I’m – the example I’m using is if a particular reporting requirement dictated by the tactics section was inadvertently put into the software selection section. There, the example I’m using is whether a particular postal code is related to an offshore isle or not. It is possible to locate that particular functionality within a software system, it certainly is, but that is a redundant piece of information, in my perspective.
So, for instance, you have a UK postal code and the tactics section might require an extra entry of checking off a box to say whether this postal code relates to an offshore isle. You certainly can require your software system to record that but the correct place to record that information is more in the data management system, because it is redundant, there is no need to introduce that into a user interface screen. It’s defined. It’s pre-defined. Everyone, you know – that is something that can be managed further down in the hierarchy.
If you did require the software to do that, here are the downfalls of that: so number 1, the extra amount of coding that it would take just to implement that one little change. But the bigger issue is you are now allowing the possibility for, internally, your data to not be correct. So you’re giving the user the option of giving a postal code and a particular category or tag to that postal code. The user might put that in incorrectly. However, if you have that pushed down into the data management section and the data management section has a definition of each postal code and whether it is an offshore isle, it’s taken that labour off the user it’s taken the labour out of creating the software and it’s also maintaining the integrity of the reporting that will be using that.
Mr Beer: In your answer there, you gave, as one of the consequences of not adopting this approach, the need for writing much more additional code: “extra coding”, I think you called it.
Charles Cipione: Yes.
Mr Beer: Is there a problem by having to write additional code and, if so, what is it?
Charles Cipione: Well, the first problem is that, if that code is not necessary, there’s going to be a cost component to that code. So that’s the first problem. But the second problem, really, has more to do with being – having internal referential integrity of the data and that’s what I just described. It is possible, if you put that in, to have data that doesn’t agree with each other. If I have a particular postal code that is related to London and I have the option of checking that off as being an offshore isle, I’ve just introduced a data error into the system.
Then the third thing would be the maintenance of that code, the maintenance of the software, to the extent it’s ever upgraded, would have to take this extra coding into account and have a knock-on effect of perhaps increasing the maintenance costs further down the line.
Mr Beer: Where does the concept of data-driven logic – something that you’re going to speak about in a moment, I anticipate – fit in with what you have just described?
Charles Cipione: So data-driven logic would be what we just described as the proper placement for this particular example. It is a reference table – in this example would be a reference table for all the postal codes and, instead of having hard-coded in as software, we could have a reference table, basically, which just basically means I have a list of all my UK postal codes and then I have an indicator of whether that postal code should be considered an offshore isle or not and the people maintaining that particular database, you know, let’s say that there was – you know, you had a new isle all of a sudden pop up. It would be much easier to maintain that reference in the data management section rather than in the software part of the hierarchy.
Mr Beer: Can we turn to systems development, please. In paragraph 3.5.1 of your report, you make a point as to the distinction between the terms “software” and “system”; could you explain that, please?
Charles Cipione: Certainly. Software I would describe as application code. Application code is what controls all the hardware but a system is a more universal charm, which includes hardware and communications and a number of other things. It’s not simply just the software – it’s not just the application code; it’s the universal system of all the components that are related. So, for instance, you know, making sure that your communication lines are working right; making sure that your printers are working right; making sure that any other pieces of hardware related to the system is included.
Mr Beer: So the system is how the software and the hardware operate together?
Charles Cipione: Yes, that is correct.
Mr Beer: A subset of that is the software, and that’s the system or the part of the system most often known as an application, that directs, in particular, the computer’s hardware?
Charles Cipione: Yes, that’s correct.
Mr Beer: You explain in paragraph 3.5.5 the nature of hardware devices. I wonder whether we could look at that, please. I appreciate that, to you, a lot of this may be very basic indeed, as indeed to a number of people listening or watching online. But I want to take it at this level right at the beginning of the Inquiry for a reason, please.
You mention a series of hardware devices and you categorise them as, firstly, input devices. Can you give some examples of those, please?
Charles Cipione: Certainly. As I say in my report, keyboards, mice, touchscreens, card readers and, in fact, even some storage devices at times will act as an input device.
Mr Beer: Secondly, you categorise some processing devices. Can you explain those and give some examples, please?
Charles Cipione: Certainly. The CPU of the computer, or the brain of the computer, is the main processing device.
Mr Beer: Storage devices, thirdly. Can you explain those and give some examples?
Charles Cipione: So hardwares, memory, like CD-ROMs, anything that retains – that persists information.
Mr Beer: Is it right that a storage device could be either an output or an input device?
Charles Cipione: That’s correct.
Mr Beer: Could you perhaps give us an example of that?
Charles Cipione: Certainly. So if you are working on a spreadsheet that perhaps you saved yesterday, you – as you pull up that spreadsheet, it’s referencing the hard drive to pull up the information that’s on the spreadsheet. So at that point in time, your hard drive is considered an input device, because that’s where your application is receiving information from.
Then, as you make changes to that spreadsheet and are done for the day and save it, that same hard drive is being saved to and, at that point, it turns into an output device.
Mr Beer: You say in your report, where you indeed give that example:
“Even in this very basic explanation, we can foretell the bleeding of meanings.”
What did you mean by that?
Charles Cipione: So what I mean by that is that, depending upon the context of what we’re talking about throughout the course of any discussion about something as complex as systems development and deployment and maintenance, you really need to understand the particulars and the details of the situation at hand to totally understand the implications of what’s going on.
Mr Beer: In paragraph 3.6 of your report, you give an overview of the different types of software. These, as you explain, sometimes interact with the hardware of a system, and sometimes they interact with other software, and sometimes they interact with the user of a system. You set out the four main categories of software. The first is an operating system or OS software. Can you, please, explain that and give some examples?
Charles Cipione: Certainly. So some examples would just be, like, Microsoft Windows or Linux or macOS, which is Apple’s operating system. Essentially, what this does is it provides the interface between the hardware and everything else that happens on the system. It’s where a device driver sits, it’s basically how – it’s the rulebook for how the hardware of that particular system is going to interact with any other bit of software.
Mr Beer: You say in your report the operating system software is the low-level software that allows the software to interact with the computer’s hardware. What do you mean by “low-level software”?
Charles Cipione: It’s the baseline software that basically is the train conductor for everything that happens on the computer and what I mean by that is it allows everything to interact with the hardware because, ultimately, you know, a computer is a piece of hardware and there could be multiple different pieces of hardware on that computer. The operating system is the level of instructions that allow the hardware to interact with anything else that is on that particular computer.
Mr Beer: The next species of software that you describe is a database management system or DBMS. Can you please explain that and give some examples of it?
Charles Cipione: Certainly. So, oftentimes, it’s needed to – systems need to collect and organise information in a structured manner and the database management system software helps to do this. Oftentimes it’s in a structure but it doesn’t always have to be – when I say “structured”, I’m really referring to like tabular formats. You can often think of a database management system as being a series of tables that hold information and can be – and have relationships to other tables.
So, like, an example would be perhaps I have a sales system and that sales system I want to know who all my customers are. So there might be a table that holds just customer information and a reference key for that customer information. But it might also have a different table that keeps track of all the sales I’ve made to that customer.
So what a database management system does is it tries to organise that information in a way that minimises the amount of space it takes to record all that information and allows me to do some analyses on that information.
Mr Beer: You give examples as Microsoft’s SQL Server and the Oracle Database. Can you explain those, what they do?
Charles Cipione: Certainly. They do exactly what I just described. So Microsoft SQL Server and Oracle Database are both examples of relational database systems, that would be the more tabular structure form and they really underpin most large, like, accounting systems and ERP systems.
Mr Beer: What’s an ERP system?
Charles Cipione: Enterprise relationship platform system that – it’s the general software that helps run an enterprise, so it usually includes your general ledger as well as any other accounting subsystems, like accounts payable, accounts receivable, you know, your inventory system. You know, any – like an SAP would be an example of an ERP system.
Mr Beer: What’s an SAP?
Charles Cipione: SAP is a brand named ERP system that basically helps run your enterprise, so it will do everything from financial to operational services for your enterprise and, in theory, is integrated so it allows all of those different systems to speak to each other.
Mr Beer: The third species that you describe is application software. Can you please explain what you mean by application software and then give some real world examples?
Charles Cipione: Certainly. So an application – that is a very general term. You know, if you asked me to sit down and create an address book for you, to keep your calendar and keep your contacts, if I programmed it for you, I would consider that a piece of application software. The SAP system that we just talked about, I would consider that a piece of application software. Usually, it is a piece of software that is built for a specific business, or maybe even non-business, purpose but it usually is custom built for a particular purpose, even things like Microsoft Word or Microsoft Excel, I would consider those pieces of application software.
Mr Beer: Even though they are not built for a specific business?
Charles Cipione: That’s right, but they are built for a specific purpose.
Mr Beer: Lastly, fourthly, you describe the fourth species: application development software. Can you please explain what application development software is and give us some examples?
Charles Cipione: Certainly. So if you were to ask me to build you, you know, a contact tracking system, perhaps I might want to use what I’m referring to here as an application development software system and what that is is it is a set of software packages that allow programmers to efficiently design, develop, deploy and maintain software. So it’s specific to systems development, design and deployment, and it supports those – you know, that effort in organising all the code, organising the releases and keeping track of that. I believe that I – you know, like Microsoft – Microsoft has a studio it’s called Visual Studio, it is an application development software, and Android also has a studio, if you wanted to –
So if I wanted to deploy a mobile app on Android, I could use Android Studio’s application development software package to help me do that.
Mr Beer: So it’s software for writing –
Charles Cipione: Yes, it’s software for writing software.
Mr Beer: – and maintaining and amending and changing software?
Charles Cipione: Exactly.
Mr Beer: You express a caveat at 3.6.2 that there are many other types of software but those four categories allow you, in this report, to illustrate how software types interact with each other?
Charles Cipione: Yes, that’s correct.
Mr Beer: You give an example at 3.6.3 and can you just explain that to us, please?
Charles Cipione: Certainly. In this example I’m talking about if we are developing an accounting application, the first thing that we would use as the developers of the accounting application would be the application development software to that. Knowing that accounting uses a lot of – or expects a lot of transactional information, I would also expect that a database management system’s piece of software would be used to help record and retain that information. Both of those would be obviously, as I said before, interacting with the operating system software and, as it was developed, all of that would be considered an application.
Mr Beer: Thank you. In paragraph 3.7 and following of your report, you explain to us the concept of the software development life cycle or the systems development life cycle, in both cases shortened to the acronym SDLC. Could you explain the difference, if there is any difference, between “software DLC” and “systems DLC”?
Charles Cipione: Certainly, much like we discussed a few moments ago, the distinction is, if it was just software, I would be concerned only about the application code here but, taking a more universal view on the topic, and I want to, as – as a system is deployed, it is not simply just a software. If I want to take into account things like hardware and communications and all of the things outside of the purview of this software, I’d want to describe it as a systems development life cycle.
Mr Beer: You focus in your report on the latter of those: the systems development life cycle; is that right?
Charles Cipione: That is correct.
Mr Beer: You explain that, although there are a variety of approaches in practice across teams, there are seven commonly used stages; is that right?
Charles Cipione: That is correct.
Mr Beer: The first of those is planning. Although that may be obvious from the word, can you explain in this context what is meant by it?
Charles Cipione: Yes. So planning, as I say in my report, this is the stage that determines what’s being requested and trying to just put together an overarching plan of how you would approach fulfilling that particular request. It is very closely joined with the next section, which is the analysis. So I would almost talk – distinguish these two as much like the strategy and the tactics of a particular development of a system.
Mr Beer: You say that analysis, secondly, is the stage where the design team gathers as much information as possible about every detail of the requested system and covers issues such as functionality, performance, equipment and cost; is that right?
Charles Cipione: That is correct.
Mr Beer: Then the third stage, design: can you explain what’s involved in that stage, please?
Charles Cipione: Certainly. The design is basically the roadmap for how you are going to achieve the goals set out in the planning and analysis stage. This includes a lot of different things. It is considering both the architecture of just the software, as well as how the – what hardware is required and making sure that the design of the software is properly accounting for the required hardware that’s associated with the system, including communications, including all of the upstream and downstream processes.
So, for instance, I might want to bifurcate or trifurcate my design into here is what the user is going to see, here is what the operational – the operations of the communications between perhaps a bunch of satellite users and a central repository of information that’s going to be collecting all of that information for users. We need to understand what is going to be – how this information is going to be consumed, what needs to be done with it. It is trying to take a very structured, rigorous approach to understanding not only what is being requested right now but also perhaps anticipating that changes might be required in the future. So kind of baking that into the structure of the way this system is designed right now to accommodate, hopefully, you know, reasonably anticipatable future requests.
Mr Beer: You say in this paragraph:
“If an external resource is determined to be appropriate, an integration portion of the design will be documented.”
What did you mean by that?
Charles Cipione: Certainly. So oftentimes, especially on large projects or complex projects, the team that is – that has assumed the role of the general contractor for a particular piece of software or a system, I should say, rather, might not need to develop every bit of technical feature from scratch. They might be aware that there are components that exist right now from people outside of their particular programming staff that – that functionality already exists.
So, to the extent that they get to a buy or make decision, they might decide that they would prefer to go out there, out to the market and purchase an existing piece of technology and incorporate that into the system that they’re developing.
If they do that, they need to be – they need to be well coordinated with that third party that is providing a particular function or a particular feature that’s going to be incorporated into the system, so that everyone knows exactly what’s expected. Everyone knows, you know, the – because there’s a lot of technical details when you’re incorporating someone else’s piece of software, someone else’s solution for a particular function of your system.
It’s very important that everyone understand exactly what’s expected from both sides so that it operates correctly when you actually fold everything together and deliver what you’re calling a system.
Mr Beer: Thank you. The fourth stage is development and you explain that using the technical design document from the previous stage, the development team will transform the design into a functioning system.
Charles Cipione: Right. So this is where it goes from theoretical to practical. This is the – once the design document has been created, it is then used as basically the recipe book for the development team to actually code the software, to do the integration of the hardware with the software that will create the system and that will include, you know, hardware such as, you know, printers or touchscreens, but as well as making sure that things like communications systems are working properly, so that all the different components of the software or all the components of the design, many of which are software, but are connected by different hardware pieces.
So the development is taking the design, which is the theoretical – the theoretical roadmap for the system and actually turning it into a real piece of – a real system which includes all the hardware and software components.
Mr Beer: To be clear, this is the stage at which code writing or coding occurs?
Charles Cipione: That is correct.
Mr Beer: The fifth stage is testing. You say:
“This phase is used to ensure that the results of the development phase align with the expected functionality, performance and hardware described by the technical design document.”
Is this phase an important one, the testing phase?
Charles Cipione: Oh, yes, of course it is. The design provides the roadmap. The development is the actual application of that roadmap to make something – to make a real piece of – a real system which includes the software. But we need to make sure it works correctly and, in order to do that, there is always a rigorous testing process that accompanies the initial deployment of the software or of the system.
Mr Beer: You explain that there are two levels of testing: quality assurance, QA, and then user acceptance testing, UAT, yes?
Charles Cipione: Yes, so oftentimes, or most of the time, the testing first is done internally by the same group that is writing the software and there’s a division of labour within that group. Usually, there are the developers – or there’s the designers. But there’s the developers, and then there is a different group within that particular firm that will test it. It’s important that they be independent of the development group for multiple reasons but the most important one is they need to have an independent view on whether the system that was created by the development group actually adheres to all of the design specs that came out of the design group.
So you have an internal team that will go through a battery of tests, it’s usually a very rigorous set of tests that make sure that everything that they see in the actual development of the system adheres to the design specifications that was given to the developers but independently verified by the testing group, by what I’m calling QA, quality assurance.
Mr Beer: You have sometimes spoken in the present tense there. To what extent was that about which you just spoke commonplace 20/25 years ago?
Charles Cipione: I would say that, as long as software has been developed, in my experience, which has been since the ’90s, that a QA function has also existed.
Mr Beer: You emphasise that this group should be a separate group of professionals but within the development and design team?
Charles Cipione: Yes. Yes, it’s not the developers but it is from the same enterprise as the developers.
Mr Beer: Yes, the same company?
Charles Cipione: Yes.
Mr Beer: You say it should be independent. I think you emphasise why that was. You say there’s a range of reasons, presumably not marking one’s own homework is one reason?
Charles Cipione: Right. I mean, practically speaking, even when you’re writing a report, it’s always good to get a fresh set of eyes on the report to see things that perhaps you’re blind to. So that’s just a practical aspect of having an independent group of people do the same thing in the context of software systems. It’s just good to get a fresh set of eyes on something.
It’s also good to have an independent group because the roles are different. The structure and rigour around a group of programmers that do testing is different than the structure and rigour of a group of programmers that do development.
Mr Beer: The second species of testing or level of testing you describe as:
“User Acceptance Testing … A small group of users from the group requesting the system then performs ‘real world’ testing to make sure the system meets their expectations.”
Can you explain in a little more detail what’s involved?
Charles Cipione: Certainly. So once a system has gained approval by the quality assurance group of testing, the first group of testers, a company would have two options. We can either roll this software out to the entire user community or we can roll it out to a very small group of users to make sure that it’s acceptable to them. The benefit of rolling out to a small group of users is to identify operational issues, is this system understandable to you, as well as to catch maybe some errors that slip through the cracks of the quality assurance.
The reason I said it was a benefit is, oftentimes, the user community and the developer community are two completely divorced communities. What the design and development team might think of as a great way to operationalise something in a system might not be as appetising to actual users of that system and if you roll it out to a small group of users in this user acceptance testing, you get the opportunity to get more stylistic feedback, as well as doing one extra level of testing to make sure that the functioning of the software or the system is performing as needed.
Mr Beer: You explain in this paragraph that often there are certain benchmarks that define whether the system can be permitted to go to the next stage, the deployment stage, ie a written down, recorded set of criteria; is that right?
Charles Cipione: That’s correct.
Mr Beer: You explain that the system does not need to be perfect to be deployed but it needs to be acceptable to the user community?
Charles Cipione: That is correct.
Mr Beer: So one will often see criteria developed and the performance and operability and functionality of the system measured against those criteria?
Charles Cipione: That is correct.
Mr Beer: The next stage is deployment. Can you explain what happens at that stage, please?
Charles Cipione: So once user acceptance testing has passed, has given the system a passing mark, it’s now time to take this system and make it accessible to the entire anticipated user community, and deployment is that process where you are now rolling out the software to the entire population of users anticipated, you know, through this process –
You know, when the software gets – when the agreement to make the software happens, you anticipate what the entire user community is. The user acceptance training – testing was a small set of it. The deployment is talking about now rolling it out to everyone, making sure that – or allowing everyone to access this particular system.
Mr Beer: You explain that this can be done in stages or all at once.
Charles Cipione: That is correct, depending on the circumstances. Sometimes it is advisable to go ahead and release this particular system to everyone all at once. Other times, maybe there are logistical issues that make it more advisable to roll this out to 10 per cent of the user community this week, 10 per cent next week, 10 per cent the week after. It just might be a logistical issue, but both deployment strategies or both deployment options are available, and it really just depends on agreement between the people contracting for that system and the people delivering the system.
Mr Beer: You say that this stage involves the delivery of documentation to users concerning the operation of the system?
Charles Cipione: Yes. So as the system is rolled out, you will then also need to make sure that the proper support for the users exists and that is done in two forms: usually a user guide and access to a help facility, meaning either, you know, a phone call to a helpdesk, an email, some sort of communication mechanism, to the extent that users do experience issues, that they have something besides the documentation. They should refer to the documentation first but, to the extent that that’s not helping them in their particular situation, they need to have access to someone else that can help them in realtime.
Mr Beer: You describe this as a contract mechanism (sic) for the system’s helpdesk. What did you mean by that, a “contract mechanism”?
Charles Cipione: A “contact mechanism”?
Mr Beer: Ah, “contact mechanism”, I misread the word. You just described it.
Charles Cipione: Yes –
Mr Beer: Please ignore that question.
Charles Cipione: – it’s how you get in touch to the helpdesk to the extent that you need the help.
Mr Beer: You included in the answer before last a mention of the need for training as part of this stage.
Charles Cipione: Yes. So depending on how complex the system is, in addition to the training manuals and the access to a helpdesk, it could require training, especially if this particular system represents kind of a paradigm shift, you know, where you’re moving a lot of people from doing something that they used to do one way or never did at all and are just not familiar with the entire concept of what we’re trying to achieve here, and how the software is – or the system is helping you achieve that.
Training is another avenue to make sure that the users are well situated to employ and utilise the system.
Mr Beer: Then the last stage is maintenance of the system when it’s in use.
Charles Cipione: Yes. Yes. So maintenance – so even once we’ve gotten to the point where the system has gone through all the testing, all the training is happening and the user community is interacting with the system, there is a possibility that the users have identified some bugs or errors in the system, in which case those bugs and errors need to be addressed. It also – usually, when a system is rolled out and to the extent that the user community is excited about the system and sees the potential of other things that the system can do, the ability for the users to communicate those desires, for new functionality, usually is collected during this point.
The maintenance, therefore, is twofold: one, if there are errors or there are bugs in the system, it’s to allow for the correction of the bugs. It’s also to act as a collection of basically wish lists of things that the system could do in the future, to the extent that everyone agrees that it’s proper to go ahead and create a different version of the system.
Mr Beer: Can we just complete this section of your report before the morning break. In paragraph 3.8.1 you describe or explain how:
“Over time, there has been an evolution of how the stages of SDLC are modelled.”
You describe, I think, the oldest model as being a waterfall concept. Can you please explain what that involved?
Charles Cipione: Yes. So in the past, a waterfall methodology was often employed, a waterfall SDLC methodology was employed which basically said I want to try to do everything in a monolithic fashion. I want to know every design aspect and get that set. I want to develop everything in – that is described in the design concept. Basically, I want to do everything in each stage and not move on to the next stage until the prior stage is complete. So that’s the old way of doing it.
In more recent times, what has happened is people or development communities have broken up the design, development and deployment into smaller chunks. So they’re not necessarily creating the entire system at once but they’re creating components of the system at once and trying to move those components – those bitesize chunks through user acceptance and – or through design, development, testing and maintenance in smaller chunks and that –
What that does is it allows kind of a trickle effect of getting the system out into the user community a little faster, although be it (sic) in smaller functional chunks than the entire system at once.
Mr Beer: You described that as Agile development?
Charles Cipione: Yes.
Mr Beer: Would something we’ve seen in the papers here called either – Inquiry papers, rather than the newspapers – rapid application development technique, be a form of Agile development?
Charles Cipione: Yes. Yes. So there’s lots of different flavours and there’s lots of different nuances but, essentially, what I’m trying to describe here is that you can – there are many different approaches to doing systems development life cycle and, oftentimes, they’re really around how quickly we want to get things through, what level of acceptance is required, maybe in a rapid level of acceptance. You don’t need it to be as perfect as in a waterfall level of acceptance.
That’s really a stylistic and taste choice on both the developer and as well as the user and that’s just something that is – there’s constant – there’s a much more frequent feedback loop in the rapid development as opposed to the waterfall method.
Mr Beer: I described it in opening as an approach to software development that focuses more ongoing software projects and user feedback and less on following a strict plan of development and testing cycles.
Charles Cipione: Yes.
Mr Beer: Does that sound about right?
Charles Cipione: That does sound right.
Mr Beer: Thank you.
On that happy note, can we break for the morning, please, sir, if it suits you. Just coming up to 11.25. Can we say 11.40 or 11.45, sir?
Sir Wyn Williams: Well, Mr Cipione, you are answering very many questions. How much of a break would you like? I’m very happy to extend the break until 11.45, if that suits you.
Mr Beer: Yes, thank you very much. 11.45. Thank you.
So we’ll break until 11.45, thank you.
(11.25 am)
(A short break)
(11.45 am)
Mr Beer: Good morning, sir. Can you see and hear me?
Sir Wyn Williams: Yes, I can.
Mr Beer: Yes, and we can see and hear you. Thank you very much.
Mr Cipione, can we turn to section 4 of your report, which starts on page 21. In this section of your report, you set out a summary of the Post Office and its branches, a summary of the services available at Post Office branches, a summary of the Horizon IT System, looking first at the components of Legacy Horizon, which you describe as components (a) to (d), then you look at the components of Horizon Online, again describing them as components (a) to (d), and then you deal with the important activities or the important concepts of remming in and rolling over.
In order to provide those summaries, is it right that you have drawn on the documents set out in paragraph 4.1.3 of your report – which I would ask to be displayed, so if you just scroll down, thank you – in (a) to (f)?
Charles Cipione: Yes.
Mr Beer: So those six documents that are listed there are the essential bases for what you say by way of summary?
Charles Cipione: Yes.
Mr Beer: You enter a caveat at the foot of the page at paragraph 4.1.4, in which you say:
“I have endeavoured to summarise these documents to what I consider an appropriate level of detail Inquiry, but this has necessarily required me to omit some of the extensive technical details …”
You explain that one document runs through 819 pages and another document runs to 417 pages.
Charles Cipione: Yes.
Mr Beer: So you have summarised but hopefully not oversimplified?
Charles Cipione: That was the intent.
Mr Beer: Can we start then with the Post Office and its branches, turn to paragraph 4.2, please. You explain that although the formal company name and structure of the Post Office has changed several times over the course of the last few decades, it’s remained, in essence, a government-owned company responsible for operating a network of branches throughout the United Kingdom in which it offers post and other services to the general public?
Charles Cipione: Yes.
Mr Beer: Between 1986 and 2001 the part with which we are most concerned was called Post Office Counters Limited, or POCL, as you describe them?
Charles Cipione: Yes.
Mr Beer: From 2001, it was known as Post Office Limited?
Charles Cipione: Yes.
Mr Beer: You explain in your paragraph 4.2.3 the three different species of Post Office branches. Firstly, Crown Office branches and you explain that these are – these branches are directly managed by Post Office Counters Limited and are known as “Crown” post offices. They’re run by employees of Post Office Counters Limited and such employees are commonly known as Crown Office employees?
Charles Cipione: Yes.
Mr Beer: The second species are agency post offices and can you explain what you understood agency post office branches to be?
Charles Cipione: My understanding is that these are branches that are located in shops or other facilities around the UK and are where Post Office services can be offered by the shopkeepers.
Mr Beer: The distinction is that the branches were owned by the subpostmasters who were agents of Post Office Counters Limited?
Charles Cipione: Yes.
Mr Beer: The third species are outreach services and you describe these as typically being small, part-time branches that may use a village hall or mobile van to provide Post Office services to communities that might not otherwise receive them?
Charles Cipione: Yes.
Mr Beer: In a graph, if we can go over the page, please, which is your figure 4.1, and if we could enlarge just the graph, please. Thank you.
We can see the changing nature of those three species of branches depicted in this figure 4.1. I think this describes how many thousands of each type of branch there were for the period 2000 to 2021?
Charles Cipione: Yes.
Mr Beer: I think, would this be right: the data shows firstly a decline in the overall number from about 18,000-odd to less than 12,000-odd?
Charles Cipione: That is correct.
Mr Beer: It would show, secondly, a decline in the number of Crown Office branches, that’s the purple on the graph?
Charles Cipione: Yes.
Mr Beer: I think you make the point in your report – it’s paragraph 4.2.5, no need to look it up at the moment – that although, certainly in 2003, the Crown Office branches represented only 3 per cent of the overall estate, the Post Office said that they accounted for over 20 per cent of transactions by volume?
Charles Cipione: That is correct.
Mr Beer: I think the third thing we can probably take from this graph is that the number of outreach services that were offered grew very substantially from 2000 up until 2021, there depicted by the dark green on this graph?
Charles Cipione: Yes.
Mr Beer: In paragraph 4.3 of your report you explain the services available at Post Office branches and you say at one time it was estimated that some 170 services were offered, and they include the well-known services listed in your seven paragraphs (a) to (g), and these are all examples of what you describe as transactions. What do you mean by the phrase “transactions”?
Charles Cipione: In the context of the Horizon System, as each one of these services were engaged upon by the customers through the Horizon System, they would generate a transaction that would need to be recorded within the Horizon System.
Mr Beer: So, essentially, a transaction in this context is any event in which a customer used a Post Office service in a branch that needed to be recorded in a system?
Charles Cipione: That is correct.
Mr Beer: You make the point later in your report, it’s paragraph 4.3.6 – no need to turn it up on the screen – that not all transactions were internal to Post Office Counters Limited; is that right?
Charles Cipione: That is correct.
Mr Beer: Is that because Post Office Counters Limited was providing services to clients, some in the public sector and some in the private sector?
Charles Cipione: Yes.
Mr Beer: Can you give some examples of services provided to public sector clients?
Charles Cipione: In 4.3.6, I describe the Driver and Vehicle Licensing Agency and the Department of Work and Pensions would have been public sector clients.
Mr Beer: Private sector clients, can you give some examples of those, please?
Charles Cipione: Camelot, British Telecom would be examples.
Mr Beer: I think you mentioned Girobank too?
Charles Cipione: Yes.
Mr Beer: That meant that some of the money that was collected in branch would need to be sent to, or indeed obtained from, such clients but that was done by Post Office Counters Limited; is that right?
Charles Cipione: That is correct.
Mr Beer: You make the point in paragraph 4.3.7 that it was important to keep a record in the branch of all such transactions so that Post Office Counters Limited could work out which clients it needed to pay money to or claim money from, as well as ensuring that its own cash and stock was accounted for; is that right?
Charles Cipione: That is correct.
Mr Beer: You explain in paragraph 4.3.8 that before Horizon was introduced, a number of branches would record their transactions in paper form in ledgers or other similar documents, or use their own electronic point of sale or EPOS systems, one of which was called ECCO+?
Charles Cipione: Yes.
Mr Beer: The ECCO+ system, is that essentially the brand name or product name of the supplier of that system?
Charles Cipione: That is my understanding, yes.
Mr Beer: When we mention transactions in this context, they do not include occasions, is this right, where a customer purchases an item in a shop that is co-located with the Post Office, like confectionary or bread and milk or a newspaper?
Charles Cipione: That’s correct. The Post Office transactions or the POCL transactions were taken care of on the Horizon counter. All of the shop transactions were taken care on – through a different method.
Mr Beer: So the transactions that I mentioned, or the type that I’ve just mentioned, would be processed separately from those of the Post Office branch, often via a separate counter?
Charles Cipione: Yes.
Mr Beer: So perhaps a number of us have experienced it. If you wanted to buy a book of stamps and a newspaper, you’ve got to get in two queues sequentially.
Charles Cipione: Yes.
Mr Beer: As we’ve seen from your table, the majority of Post Office branches were agency branches. They were owned and managed by subpostmasters –
Charles Cipione: Yes.
Mr Beer: – and the cash and the stock was owned by Post Office Counters Limited but managed day-to-day by the subpostmasters?
Charles Cipione: Yes.
Mr Beer: Can we turn, please, to the Horizon System and turn up paragraph 4.4.1. Thank you.
You explain that the system was introduced in stages, known sometimes as a rollout, between 1999 and 2000, and that the objective, as you understand it, of the Horizon IT System implementation, was to modernise the point of sale and managerial accounting functions across the network of Post Office branches. Today we might describe this process as “digitising” the branch network?
Charles Cipione: Yes.
Mr Beer: You explain that the Horizon System is still in use today, albeit it’s gone through the three main iterations that we have previously discussed in its 22-year or so lifetime?
Charles Cipione: Yes.
Mr Beer: Can we begin with the Horizon System and we’re going to call it Legacy Horizon, as it came to be known. Turn over the page to paragraph 4.5 and the table at 4.1. Thank you. You kindly set out a brief history of Legacy Horizon in this table at 4.1. Can we just run through it, please, so that we’ve got the larger milestones in mind at this early stage of the Inquiry, please. Again, this is extracted from the document set that you mentioned earlier on; is that right?
Charles Cipione: Yes, that is correct.
Mr Beer: So if you can start, please, using this table to narrate these ten or so developments in the history of Legacy Horizon?
Charles Cipione: Certainly. So, as you can see the first entry, May ‘96, the DSS and POCL jointly awarded the contract for – to ICL Pathway for what we’re calling Horizon, although you can see that there are a number of different variations of that name in here; Pathway Project, Pathway Horizon and so on. It was – ICL Pathway at the time was a wholly-owned subsidiary of ICL. Fujitsu acquired 80 per cent of ICL’s shares in 1990 and purchased the remainder in 1998, and ICL was fully integrated into Fujitsu in 2002 and was renamed Fujitsu Services Limited.
Mr Beer: Just before moving on there, this contract, the May ‘96 one, was a contract to develop an IT system that would firstly replace the existing paper-based method of paying Social Security benefits and, secondly, automate the entire national network of post offices; is that right?
Charles Cipione: Indeed.
Mr Beer: Yes. Can you move on to September 1996, please.
Charles Cipione: Sure. In September ‘96 was the Initial Go Live that was implemented in ten Post Office branches and this was an interim – a system for Child Benefit payments and was limited to that functionality.
In November ‘97 that system was extended to 200 Post Office branches, still just remaining – the functionality just being the Child Benefit payments and it was noted in my documentation that the deadline for completion of the operational live trial of the IT system, was missed by ICLPL.
Mr Beer: That’s, at that time ICL Pathway Limited?
Charles Cipione: Yes.
Mr Beer: Thank you.
Charles Cipione: In March ‘98, an interdepartmental working group was established to review the viability of the Pathway Project and the consequences of cancellation. The working group comprised officials from Treasury, Cabinet Office, Department of Trade and Industry and the DSS.
In July 1998, the interdepartmental working group reported that the Pathway Project remained feasible but required successful renegotiation of the contract with ICLPL.
In October 1998, attempts to renegotiate the terms of the contract between DSS, POCL and ICL failed. In May 1999 the original PFI contracted awarded to ICLPL by DSS and POCL was terminated. DTI announced a new partnership agreement between POCL and ICLPL.
In July 1999 POCL and ICLPL agreed a fixed payment contract to automate the national network of post offices and, in late 1999, the rollout of Horizon occurred or commenced.
Mr Beer: You mentioned earlier in your evidence this morning one of the two stages of testing or levels of testing was UAT, user acceptance testing.
Charles Cipione: Yes.
Mr Beer: So far as you know, would that refer to the stages on this table of September ‘96 and November ‘97?
Charles Cipione: Yes, that would be the user acceptance testing, you’re correct.
Mr Beer: Thank you. We can take that table down, please. Can we turn to the functionality of Legacy Horizon. We are moving to paragraph 4.5.2 of your report. You explain that there are essentially two elements to it. The first of which is the electronic point of sale or the EPOS element. Can you explain what the purpose and the function of the EPOS was?
Charles Cipione: The purpose and function of the electronic point of sale component of Horizon was simply to capture the transactions that occurred at the branches throughout the network.
Mr Beer: So it included the purchases of Post Office products, such as stamps and stationery, made by customers in branch; is that right?
Charles Cipione: That’s correct.
Mr Beer: Also transactions carried out in branch for the purposes of products or the use of services provided by clients of the Post Office, and the clients here are the things you’ve mentioned already – or the organisations you’ve mentioned already: some public sector clients, DVLA, DWP; some private sector clients, banks or Camelot.
Charles Cipione: That’s correct.
Mr Beer: You explain, secondly, that the purpose and function of the Horizon IT System was one of management accounting. Can you explain what that is, please?
Charles Cipione: Certainly. So the transactions that were collected at each one of the branches for – throughout the network needed to be consolidated and organised for purposes of doing all of the managerial accounting. What I mean by managerial accounting is I would consider the transactions operational – details of the operations of POCL’s agents, as well as their Crown Offices.
All of those transactions needed to be organised in order for POCL to do their own internal accounting as well as exchange information with all of their clients, so the managerial accounting was a step in that process to collect all of the transactions and manage them in order to supply further processes that needed to be done for their own internal financial accounting as well as to exchange information with all of their client partners.
Mr Beer: Thank you.
You explain in paragraph 4.5.3 that, in terms of the size and scale of the data process and the code written, both were substantial.
Charles Cipione: Yes, they were.
Mr Beer: You tell us that in 2003, the Post Office stated that Horizon processed nearly 2 billion transactions per annum?
Charles Cipione: Yes.
Mr Beer: But despite that, you say that it was a relatively simple task, computationally?
Charles Cipione: Yes, each individual transaction or – there weren’t any complex calculations associated with any of these transactions but there were a vast number of transactions.
Mr Beer: You say that it’s no more complex than systems operated by, for example, banks?
Charles Cipione: That is correct.
Mr Beer: You refer to an estimate that Legacy Horizon had over 3.5 million lines of programming code. What’s the general approach that a system designer ought to take to writing code, in terms of its volume?
Charles Cipione: It’s all – less is always better, certainly. However, the requirements for different systems required different volumes of software code. But less is generally a better rule than more.
Mr Beer: Why is less better than more?
Charles Cipione: Maintenance. Well, number 1, simplicity of the coding aligns with a good structure of code. But, just as importantly, to the extent that maintenance needs to be done on the code, the less code that exists to begin with, the less code there is to maintain as updates are made to the code. It’s just simpler: the smaller number of lines of code, the easier it is to maintain.
Mr Beer: Is it possible to say whether this is a high number or a low number or an average number of lines of code, or can one not apply such descriptors to it?
Charles Cipione: On the face of it, this looks like a very large amount of code. However, I have not looked at the code. I don’t know exactly what this code represents. So I don’t have an opinion whether this is an appropriate amount of code or not.
Mr Beer: Just give us a comparison. The systems that you mentioned earlier that you designed for General Motors and WorldCom, how many lines of code would we be talking about there?
Charles Cipione: I would say 20,000 lines or less of code.
Mr Beer: But, as you said, the number may be an indication that it was written as economically as could be but is a reflection of the number of tasks that needed to be performed, or it’s an indication that the code was not well written. But you haven’t subjected the code to forensic analysis –
Charles Cipione: That is correct.
Mr Beer: – because that wasn’t within your instructions?
Charles Cipione: That’s right.
Mr Beer: You tell us that the documentation ran to more than 100,000 pages. What do you mean by “the documentation”?
Charles Cipione: So documentation around the Horizon System, there was a lot of it. So documentation could be user documentation, it could be updates to user documentation, it could be technical documentation, updates, it could be business processes; all of those are encompassed in this count.
Mr Beer: This may sound a silly question but is that a high number? Does it appear to be a high number?
Charles Cipione: It appears to be a high number to me. We need to take into account versioning, though. I’m positive that this probably encompasses, you know, version 1, version 1.1.1, you know, of all of the different dimensions of documents. So it does appear to be a high number but I have not catalogued it or made a determination whether it is excessive or not.
Mr Beer: You make the point that the system was created specifically for the purposes of servicing the Post Office branches and didn’t have the added burden of integrating existing technologies.
Charles Cipione: That’s correct.
Mr Beer: Would that be a limitation on the possibility of additional complexity of a system?
Charles Cipione: It would indicate that the complexity of the system was completely defined by this process and not aggravated by any environmental factors of an existing system.
Mr Beer: In paragraph 4.5.5 you say that the project was “ambitious” in both “scale and scope” and you draw some contrasts with the state of information technology “at this time”, ie from about ‘96 to about 2000.
You remind us that the – or remind some of us that the Nokia 3210 was the best-selling phone of 1999 – some of us would wish that that technology still existed – but it had a monochrome screen; is that right?
Charles Cipione: Yes.
Mr Beer: It didn’t have any touchscreen navigation; we had to wait until 2007, I think, for that –
Charles Cipione: Yes.
Mr Beer: – and one couldn’t access the Internet through a browser on the phone?
Charles Cipione: That is correct.
Mr Beer: You tell us that at this time, only about a third of people were estimated to have a personal computer –
Charles Cipione: Yes.
Mr Beer: – and only 30 per cent of adults had access to the Internet?
Charles Cipione: Yes.
Mr Beer: We had to wait until 2004 for all of the benefits of Facebook?
Charles Cipione: Facebook arrived in 2004.
Mr Beer: At this time, the IT world was focused on the so-called Millennium Bug?
Charles Cipione: Yes.
Mr Beer: In terms of IT development, you tell us again here that the prevailing method was the waterfall method, and Agile development wasn’t mainstream in IT development at this time?
Charles Cipione: That is correct.
Mr Beer: Can we turn with that background to the seven elements or aspects of the development and implementation of the Horizon System, which drove its complexity. There set out in your paragraph 4.5.6, which is at the foot of page 28 and on to 29, and there are seven elements set out in (a) to (g), can you talk us through those? Firstly, the need to design a system that connected all Post Office branches to a central server but could also withstand a loss of connectivity?
Charles Cipione: Yes, so this was – you know, at that time, this was a much more difficult problem than it is now. Simply because our communications infrastructure is much better now. It’s much more robust. The reliability of connectivity, including the expense related to that connectivity at this point in time, really provided issues to anyone trying to maintain what I would refer to as a client-server type process, meaning you have satellite systems which were the clients and you would have the central system which was a server.
This is just more a talking term. I’m not positive I would describe the Horizon System as a client-server but it’s a good set of words to use in describing it.
The fact that they had to contemplate – they being ICL Pathway – had to contemplate an extended loss of connectivity meant that they had to put in guard grills and safety nets for those circumstances where the – where they knew the connectivity wouldn’t exist. So they needed to not only create a design that allowed for a system that is connected to work but they also needed to design – they needed to anticipate the fact that they could not be connected.
So those were two different logistical issues that they had to incorporate in their design and development of the system.
Mr Beer: The point you’re making here is that the system needed to be designed so that it could maintain its functionality or most of its functionality whilst there was a lot of connectivity, i.e. customers could still be served in the branch.
Charles Cipione: That’s right, because it is – the customer at each one of the branches did not want a connectivity issue – I’m sure didn’t want a connectivity issue to stop them from purchasing stamps, for instance. So that required the need for the design to anticipate connectivity issues.
Mr Beer: And then allow for correct synchronisation once connectivity had been restored?
Charles Cipione: Right. So that’s, you know, that’s – that complicates the design and development of the system. So when you’re anticipating a loss of connectivity, you have to have plan B. Okay, what does the system do now that I know I’m not connected? Now I need to keep a persistent store of the – you know, I need to, number 1, identify that I’m not connected and, number 2, then I need to collect information until I know I’m connected again and then, number 3, when I am connected again, I need to make sure that the information that’s been stored up gets transmitted correctly to the central server.
So those are – that might sound simple, that is not a simple process, necessarily.
Mr Beer: The second area of complexity you mentioned is the need to integrate a variety of software and you mention, in particular, Riposte and Tivoli. Can you explain what riposte and Tivoli were?
Charles Cipione: Certainly. As I mentioned earlier, oftentimes in the design of a system, you would decide whether to buy or make certain functions within your system. In these instances this a buy: I want to buy. So the Riposte was a software that basically allowed for the look and feel of the counter to be pre-made, you know, the touchscreens, and all that, that was a product that was already –
Mr Beer: The user interface?
Charles Cipione: The user interface, yes. So Riposte provided that. It also provided the mechanism for capturing and transmitting the transactions that were related to all the activity that happened on the user interface through the counter. Tivoli was more of a behind the scenes type product. It was more of an operational type product but it assisted the system to update software packages and update reference data, which we’ll talk about further on in my report but it was more of an operational assistant to help the Horizon System work properly.
Mr Beer: You also say that there was a need to integrate a variety of hardware, including touchscreens, printers, communications equipment, barcode scanners, weighing scales, PIN pads, and the like?
Charles Cipione: Yes. There was a particular set up that was in the design spec and those are the hardware components that were aligned with that set-up.
Mr Beer: The third area of complexity that you mentioned is the need to accommodate hardware failures because hardware components in the 1990s were not as reliable as they are today?
Charles Cipione: That is correct.
Mr Beer: The fourth element you mention is a large and diverse user base amongst subpostmasters and the staff that they employed, which would have included varying levels of comfort using ‘modern’ IT systems, in inverted commas; is that right?
Charles Cipione: That is correct.
Mr Beer: So you’ve got a cohort of people that are more or less familiar and more or less happy with information technology at the point of rollout?
Charles Cipione: Yes.
Mr Beer: You kindly note that Fujitsu itself noted that training was provided to 63,000 staff from the ages of 16 to 87 years of age with various skills involved, and you say that would, you believe, have presented a significant training rollout and support challenge?
Charles Cipione: Yes.
Mr Beer: The fifth area of complexity you mention, I think, is the volume of the rollout and you say that, between August ‘99 and December 2000, over 14,000 branches had Legacy Horizon installed?
Charles Cipione: Yes.
Mr Beer: You subsequently, in your report, set out in the table at 4.2 – no need to turn it up – the progression of that rollout, month by month, between August ‘99 and December 2000?
Charles Cipione: Yes.
Mr Beer: The sixth area of complexity you mention was the physical challenges of installing bulky IT hardware into branches. Can you expand on that a little bit, please?
Charles Cipione: Yes, so the – there was a hardware specification that went along with the Horizon System which included the counter printers, tape rollers, card readers and whatnot. The branches might not have had space for those and that presented logistical issues for – I mean, just the physical logistical issue to implement the Horizon System at a branch. If they didn’t have space, they had that issue.
Additionally, there were communications constraints at some of the branches. Some of them didn’t have access to some of the communication systems that the Horizon System was designed for.
Mr Beer: So some of them didn’t have an ISDN line; is that right?
Charles Cipione: That’s correct.
Mr Beer: So they had to use a satellite link?
Charles Cipione: Yes.
Mr Beer: Lastly, seventhly, you mention a complexity that was added because of the need for the system to be very secure because, after all, it dealt with transfers of money as well as containing personal information?
Charles Cipione: That is correct.
Mr Beer: You say, overall, that those challenges, in your view, made the design, build and rollout of Legacy Horizon very ambitious?
Charles Cipione: Yes.
Mr Beer: Can we turn, then, to the high-level design of the Horizon System. This is over the page at paragraph 4.5.8. So bearing those points of complexity in mind, can you explain to us the elements of the high-level design of the Horizon System, starting with the fact that it was a system that used data-driven logic rather than dealing with prices in its source code; is that right?
Charles Cipione: That is correct.
Mr Beer: Can you explain this concept to me, the public and the Chair, using the example that you give of hammers screwdrivers and pliers costing £5, £7 and £6, respectively, that you have included in your report, please?
Charles Cipione: Absolutely. This is a very simple example, certainly, but hopefully the concept will resonate as you think about the more complex features of the Horizon System. But what I’m trying to juxtapose is, to the extent that we wanted to process a transaction for a hammer, screwdriver and pliers through two different paths, one path being source code path and one path being a source code supported by reference data path.
In 4.5.12, what I’m attempting to do is to show what source code might look like if it was the only arbiter of processing this data. There would have to be –
Mr Beer: So if you could walk us through the example that’s emboldened.
Charles Cipione: Certainly. I’ll just go line by line and, if you have questions, then you can ask them.
So the purpose of both of these sets of code is to calculate a basket total for the purchase of three items. So the first thing, the first function that needs to happen is we need to set our total basket to zero. We need to start at zero. Then we are going to check if the product that’s being – if one of the products that’s being purchased is a hammer. If that is correct, then I’m going to multiply the quantity of hammers by £5 and add this to the total basket amount, and you’ll notice that this is what’s referred to as hard coding.
So this is hard coded software. So the – no matter how many hammers come through here, they’re always going to be multiplied by £5 if this source code remains the same.
The next item I’m looking for is a screwdriver and, if there are screwdrivers, I’m going to take the quantity of screwdrivers and multiply them by £7 and add that to the total basket amount. Then, finally, we’re going to look to see if the product is a pair of pliers. If we do have a product being purchased as a pair of pliers, we’re going to multiply the quantity of pliers by £6 and add that to the basket.
That will generate the total basket amount based off this hard cod – you’ll notice at the bottom I also checked to see if there are any products that are not hard, you know, a hammer, screwdriver or pliers. That’s just a general error check that is commonly used in code.
But the purpose of this is just to multiply the number of hammers, pliers or screwdrivers by their respective costs or purchase amounts.
Mr Beer: You’ve written this out, I think, in pseudo-code, not the actual code that would be used. Is pseudo-code a plain language description of the steps that might be taken in an algorithm or another type of –
Charles Cipione: This is supposed to be a plain language representation of the logic that would be then implemented in a particular language that you’re using but it is not language specific. It’s just – it’s supposed to represent the logic.
Mr Beer: So and this is intended for human reading rather than machine reading?
Charles Cipione: Exactly.
Mr Beer: Now, this code enables the sale price of any of the three items to be changed; is that right?
Charles Cipione: Absolutely. We could always go in and change the pound amount that’s associated with each one of these items.
Mr Beer: But that would require a change to the source code?
Charles Cipione: Yes. That is not ideal.
Mr Beer: Can we compare this to a data-driven logic approach and look at the code that is written in pseudo-code under paragraph 4.5.15?
So, again, the part in bold and italics, under 4.5.15, if we could just blow that up.
Charles Cipione: I do want to reference table 4.3, which is behind it. The first part of this relies on the reference data that’s in table 4.3.
Mr Beer: Okay, yes, so that’s it. We can see both of those, I hope, at the same time.
Charles Cipione: Yes.
Mr Beer: So if you can talk us through this code by reference to the table at 4.3.
Charles Cipione: Sure. So, as before, setting the total basket amount to zero, and then I am iterating through the different items that are purchased. So for every product purchased, I’m going to first look in that table, in the table 4.3, to see if I find that particular item. So, for instance, if I’m looking for a hammer, I see that there’s a hammer in that table and I can see that the price for that hammer is £5. So if I find that product I’m going to multiply the quantity by the price, and add it to the basket.
In a similar fashion, when I get to the screwdrivers, I’m going to take the quantity of screwdrivers and multiply it by the £7 that’s associated with the screwdrivers and then do the same thing for the pair of pliers. I’m going to multiply that by the £6 for the pliers by the quantity that was purchased. Each time I do that, I’m adding it to the total basket amount, and, at the end of it, I should have come up with the same total that the prior version came up with.
Mr Beer: But the difference here is that any price changes can be made by an alteration to the product master table with no need to fiddle with the code?
Charles Cipione: That’s exactly right.
Mr Beer: Is there anything else you want to say or you say in your report here that, since price changes can be frequent, it is appropriate to use the latter method rather than the former?
Charles Cipione: Yes. So data-driven logic or data-assisted programming allows for adroitness in maintaining the code, or maintaining the system, because you are not required – whenever you make a change to source code, and try to deploy that, you should be going through the testing process to do that. To the extent that you can remove items like price changes from the code to more of a data driven technology, that reduces the amount of items that need to be tested. Because you already know that the code works. You simply need to make sure that you are maintaining that table correctly without requiring going through the whole testing cycle for any new code.
So, in the first example, if we changed prices, I would have to go change the code and then, in theory, I should have to test that code again to make sure the works right. In the second example, I simply need to make sure that someone is in charge of maintaining this table correctly. I’m never changing the code, therefore I don’t have to go through the testing process just because there’s a price change.
Mr Beer: In the first example you said that you would, in theory, have to go and retest. Is that because the alteration to the code may have unintended consequences for other parts of the operation of the system?
Charles Cipione: Yes. Now, in this simple example – this is such a simple example that I couldn’t imagine what this change could do but there are many more complex issues that can be handled by data-driven logic and those certainly have the opportunity to introduce more issues, or more opportunities for error, and would require a full battery of testing every time the code changed.
Mr Beer: You’ve explained to us the high-level design of the Horizon System. Can we turn to the high-level structure of the Horizon System.
You explain that there are number of ways in which you might approach the description of a system like this but you have, for simplicity, characterised the system into four main components. Do we see those listed in 4.5.17?
Charles Cipione: Yes.
Mr Beer: Can we go over the page, please, to figure 4.4, and just blow up that figure so it takes up the page. Using that figure, which sets out the component elements of Legacy Horizon, deal with the four components in summary form first, and then go into depth on three of the four components.
So starting from the bottom of the table, please, component A. Is component A or does component A consist of the counter peripherals, so the parts of the system that were located in the branch, consisting of both hardware and software?
Charles Cipione: Yes.
Mr Beer: Then moving up to component B on the left-hand side of the table, is that the communications network?
Charles Cipione: Yes.
Mr Beer: This is, in summary, functionally the same as what we’re used to nowadays, an Internet connection but, in fact, back then was either the dedicated ISDN line that we spoke about or sometimes a satellite link?
Charles Cipione: That is correct.
Mr Beer: Can you just explain what an ISDN line was?
Charles Cipione: It effectively was – it was a communication mechanism, a piece of hardware that was offered by telecom companies that allowed a connection to be made to push data through from satellite offices to a central office. It was a communication mechanism.
Mr Beer: By satellite offices, you don’t mean offices with a dish –
Charles Cipione: Sorry –
Mr Beer: – you mean remote branches?
Charles Cipione: Yes, branches.
Mr Beer: Thank you. This element, component B, is not something we are going to explore further, other than to say that you understand that the communications network was provided by a combination of services given by BT and a company called Energis; is that right?
Charles Cipione: That’s correct.
Mr Beer: Moving to component C on the right-hand side of the table here, the messaging system, did this comprise, again in summary form for the moment, the software and protocols responsible for encapsulating data and for permitting communication between branches and the Horizon campuses, as you call them?
Charles Cipione: Yes.
Mr Beer: Then lastly D, the campuses, the Horizon campuses. Can you explain in general terms, in summary level at the moment, what the campuses are and why they’re called campuses?
Charles Cipione: They’re called campuses because there were two of them, one at Bootle and one at Wigan. They were data centres operated by ICL and they acted as the collector and manager of all of the transactions that were generated at the branches.
Mr Beer: Before looking at components A, C and D in a little more detail, you make the point in paragraph 4.5.19 that the system was designed to operate with an available network connection, i.e. in an online mode, but was also designed to operate without such a connection, an offline mode. This is something you mentioned 15 minutes or so ago. Were there exceptions to that which prohibited the system from operating other than in online mode?
Charles Cipione: The – most services were able – or most transactions were able to be conducted in both the online and offline mode with the exception of two: the national banking services –
Mr Beer: The network banking services?
Charles Cipione: Sorry, network banking services and the debit card services. The reason these two services were not allowed to operate in the offline mode is there had to be a handshake confirmation that the transactions related – that a confirmation that these transactions were allowed by the actual clients before they could be transacted. So, in other words, if I was trying to withdraw money from a bank, the bank needed to tell the Horizon counter that they gave permission to withdraw that money, to the extent that the counter wasn’t in offline mode, they could not communicate with the bank, therefore that service was not available.
Mr Beer: Thank you. Can we turn, then, to the first element of Legacy Horizon, component A. Can you please run through the main physical components of the IT system within component A located within the branch. This is 4.5.21 of your report, please?
Charles Cipione: Certainly. First, there was the counter. This was the PC that had the touchscreen on it, that the SPMs and their employees would have operated the Horizon System through. So that encompasses both A and B subparagraph on this section. They had a keyboard, a barcode reader, weigh scales for weighing postal items, a tally roll printer, PIN pads and an A4 printer.
Mr Beer: Just looking at those in some more detail, the keyboard: you describe this as a specialised financial keyboard with a magnetic strip reader and smartcard reader on it. So this was a bespoke design?
Charles Cipione: Yes, that is correct.
Mr Beer: The tally roll printer, what is meant by “tally roll”?
Charles Cipione: It was the printer used for printing out customer receipts as well as some of the reports that were designed by the Horizon System.
Mr Beer: You say that some branches had but a single counter, and by a “counter”, do you mean the elements that you’ve just described?
Charles Cipione: Yes.
Mr Beer: About 46 per cent of all branches had a single counter and I think you tell us that the figures you have seen suggest that 33 per cent of branches had two counters and the remainder three or more counters; is that right?
Charles Cipione: That is correct.
Mr Beer: Is it right that, in order to use a counter, an SPM, a subpostmaster, would need to log in to the counter using their assigned username and password?
Charles Cipione: Yes.
Mr Beer: Can we look, please, at the figure on page 35, 4.5, and just blow up the figure at the top of the page there. Thank you. Is this right: your depiction of a set-up in branch where there existed a single counter?
Charles Cipione: Yes.
Mr Beer: Can you just talk us through it, please?
Charles Cipione: Certainly. So you’ll see the components that we already described. You have the counter, the PIN pad, the weigh scales, the monitor, the keyboard, the tally roll and the barcode reader and the A4 printer. All of those are connected to the PC or what we’re going to refer to as the counter, specifically. The counter – you’ll also notice that the counter, it’s called the Gateway PC and this will make more sense when we get to the multi-counter description.
But every branch had a Gateway PC, to the extent that the branch had a single counter, that single counter was the Gateway PC. The purpose of the Gateway PC designation is that is what communicated with the campuses and you’ll notice that you see that the two-direction arrow that connects the Gateway PC and the LHITS campus. Some of the components were used by the customer, some components were used by the SPMs and the weigh scales could be used by either.
Mr Beer: Can we turn to the position on multi-counter branches, please. Can we look over the page, please, at table or figure 4.7, and blow that up, please. Is this your depiction of the position in multi-counter branches?
Charles Cipione: It is, yes.
Mr Beer: Again, can you talk us through it, please?
Charles Cipione: Certainly. So the big distinction between the single-counter branch and the multi-counter branch is the fact that there are multiple counters at the multi-counter branch but the difficulty or the complexity that this presents for this particular branch is that there there’s still only one of the counters acting as the gateway between the LHITS campuses and the branch.
All of the other counters that are not considered the Gateway PC have to be connected to the Gateway PC and you can see that there is an extra box in this diagram that is labelled “Hub connecting the Counters”, and that is the extra piece of technology that needed to be introduced to connect – make sure that all the counters could communicate with each other and, importantly, communicate with the Gateway PC because the Gateway PC counter was the communication hub to the LHITS campus, which is important because that’s what transmitted all the transactions.
Mr Beer: There was a function or a feature of this system, I think, that allowed a subpostmaster to transfer an open session when dealing with transactions between counters; is that right?
Charles Cipione: That is correct.
Mr Beer: Can you just explain what that feature was?
Charles Cipione: So, the – this goes back to anticipating, you know, problems or operational situations where perhaps an SPM or one of their clerks had started a transaction with a customer at a particular counter and, for whatever reason, needed to switch to a different counter. The functionality that you just described is part of the design of the Horizon System, in that you could move that particular session from counter A to counter B, if you found that necessary.
Mr Beer: Thank you. Can we turn to the software, please. That table can be taken down. I think it’s right that Horizon used the Windows NT operating system, “NT” meaning “new technology” – is that right –
Charles Cipione: That is correct.
Mr Beer: – which, of course, it was at the time. You explain that users were prevented from directly accessing Windows; is that right?
Charles Cipione: That is correct.
Mr Beer: What’s the importance of that preventative step?
Charles Cipione: The – all of the functionality that the SPMs at the branch – SPMs and clerks at the branches, all of the functionality that they needed was through the Horizon software. There was no need for them to ever get to the operating system level, which is what Windows NT was. The set-up configured each one of the counters so that as it booted, the Horizon System came up or the repost that – the screens, the user interface, the touchscreens, would come up, and there would be no ability for the user to get to Windows.
Why is that important? If all of the functionality that the SPMs needed to operate the Horizon System existed within the Horizon application, there is no need to go anywhere else because only bad things could happen at that point. If you had access to the operating system, you could change configurations, you could do a lot of things that would deteriorate the PC and perhaps make the Horizon System not work correctly. So I suspect that is why no access to the operating system was part of the design.
Mr Beer: So the system was configured to prevent subpostmasters having access to a dot prompt?
Charles Cipione: Yes, or – yes.
Mr Beer: Is that what you (unclear: overspeaking) a little bit?
Charles Cipione: You could say the dot prompt, yes. It would restrict their access to the dot prompt.
Mr Beer: Thank you. The Windows NT operating system, how old was that at the time, in 2000, can you recall?
Charles Cipione: It was aging by the time the 2000s came around. I believe it was introduced in the mid-’90s, so it was mature.
Mr Beer: Do such operating systems have planned obsolescence within them?
Charles Cipione: Yes, they do.
Mr Beer: Do you happen to know what the planned data obsolescence was for Windows NT?
Charles Cipione: I believe that that was the mid-2000s that the planned obsolescence for Windows NT was.
Mr Beer: You tell us that rather than being allowed directly to access Windows, subpostmasters, when they logged on, were sent automatically to a piece of software that had been specifically configured for the Post Office, The Riposte Desktop; is that right?
Charles Cipione: Yes.
Mr Beer: Can you explain to us what The Riposte Desktop was?
Charles Cipione: The Riposte Desktop was the user interface that the clerks at the branches would have interacted with to operate the Horizon System.
Mr Beer: You say this was largely based, ie that system, the Riposte Desktop, on a commercial product named Riposte from the Escher Group. What do you know about the Escher Group? Who were they?
Charles Cipione: I know they were a software development group that specialised in retail software.
Mr Beer: They were a separate company from ICL Pathway Limited and Fujitsu Services Limited; is that right?
Charles Cipione: That is correct.
Mr Beer: You go on to say that the counter user interface, or UI, user interface, was designed to be as simple and intuitive as possible, and was specifically tailored for use in a retail environment. So the counter user interface, that’s the same thing as The Riposte Desktop?
Charles Cipione: Yes.
Mr Beer: You say that the intention was that the subpostmaster or clerk had no requirement to type; is that right?
Charles Cipione: That is correct.
Mr Beer: So what was done instead of typing?
Charles Cipione: There’s a touchscreen. They could use the touchscreen as well as the card readers and the PIN pads to enter information into the Horizon System.
Mr Beer: Some transactions, is this right, were initiated not by touching the screen but by an activity on a peripheral?
Charles Cipione: That’s correct.
Mr Beer: Such as swiping a magnetic card or reading a barcode using the barcode reader?
Charles Cipione: Yes.
Mr Beer: Can we display over the page, please, figure 4.8. Can we blow up that at the top of the page, please. Is this a screenshot of the user interface on Legacy Horizon?
Charles Cipione: Yes.
Mr Beer: Can we see that it’s split into two parts about two-thirds or four-fifths away across the page, from left to right. We can see a line going up and down the screen. Is that the division between the left-hand and the right-hand part?
Charles Cipione: Yes.
Mr Beer: There are a series of menu buttons on the left-hand side. Were those menu buttons or tiles available to press in the context of a particular transaction?
Charles Cipione: Yes.
Mr Beer: Is it right that some of them sometimes displayed a stop sign, preventing them from being depressed and actioned because they weren’t available for that particular transaction?
Charles Cipione: Yes.
Mr Beer: Then, on the right-hand side, is what you describe as a “stack”, showing the purchases made by the customer; is that right?
Charles Cipione: That is correct.
Mr Beer: Is there anything else you wanted to say about the user interface, the screen on Legacy Horizon?
Charles Cipione: Um, you know, the intent and, you know, whether this intent was fulfilled, is to the taste of the user. The intent was to make it as simple as possible, make it a graphical interface and to reduce the chances for error. But beyond that, no, I don’t have anything else I want to say about this screen.
Mr Beer: Thank you. You speak in your report of the concept of a stock unit, and you describe it as being an important concept. Can you explain what a stock unit is and why it is important?
Charles Cipione: Certainly. So for the managerial accounting at each one of the branches, a stock unit was a concept or an abstraction of how to – it gave the branches the ability to organise themselves however they felt fit. The stock unit – there could have been one – so we’ll get to this when we get to the rollover but, essentially, a stock unit was – could represent a particular till. It could represent a particular till for a particular amount of time. But essentially, it was a way to account for your stock and your cash at that particular branch for a particular period of time and, like I said, it allowed the SPMs to divide up their stock and cash amongst their different clerks, as well as to limit the amount of time, you know, cordon down the time that that stock and cash was in the possession of someone.
It would appear I’m not being clear here. It’s an abstract concept that basically allows you to divide up the cash and stock within a particular branch as the SPM felt fit, either across people or across time dimension.
Mr Beer: Thank you. You say in your report that it’s a way of managing cash and stock. They can be allocated to an individual subpostmaster on a medium-term basis or to an individual counter clerk, and then that person is responsible for ensuring that the stock unit balances at the end of the period, whether that be a week or however long the stock unit is allocated to them for.
Charles Cipione: Yes.
Mr Beer: You say stock units are assigned identifiers, such as “DD” or “AA”. Can you just explain what you mean there?
Charles Cipione: Certainly. So if there are multiple stock units that are being issued by the SPM, they need a unique identifier. “DD” and “AA” are just examples of how you might uniquely identify a stock unit.
Mr Beer: You say that subpostmasters can transfer stock between stock units using a function on the counter. Stock units can be individual or can be shared between multiple counter clerks. In some circumstances, the subpostmaster may choose to allocate a stock unit to a certain specified stock, such as Lottery Scratchcards; is that right?
Charles Cipione: Yes.
Mr Beer: To what extent was this left to the subpostmaster to determine, ie the object of the stock unit, lottery Scratchcards; the duration of the stock unit, a week, two weeks; or the number of people involved, a specific counter clerk or a specific counter?
Charles Cipione: This, the concept of the stock unit, theoretically provided a lot of flexibility for the SPM to use at their discretion.
Mr Beer: Thank you.
Sir, it’s a couple of minutes before 1.00. I was about to move to the topic of modes. Can we pick that up at 2.00, please?
Sir Wyn Williams: Yes, of course. We will see everyone at 2.00. Thank you very much.
Mr Beer: Thank you, sir.
(12.58 pm)
(The Luncheon Adjournment)
(2.00 pm)
Mr Beer: Good afternoon, sir. Can you see and hear me?
Sir Wyn Williams: Yes, I can. Thank you.
Mr Beer: Thank you very much.
Mr Cipione, we were just turning to paragraph 4.5.30 at the foot of page 37 of your report, where you speak about modes, ie the modes in which the counter could operate. Could you describe the various modes that you speak to there, please?
Charles Cipione: Certainly. So the concept of a mode allows the different functionality of the Horizon System to be accessed by the user, the one that everyone will be most familiar with is serve the customer, conducting the business of selling stamps, performing the transactions of the Post Office stock and cash.
Mr Beer: Just stopping there, if we go back to the screenshot of the user interface at the top of the page, please. If we just see there as this user interface is displayed in this screenshot, it is, in fact, displayed in the mode of “Serve Customer”?
Charles Cipione: That’s correct.
Mr Beer: We can see that on the tile at the top left-hand – I keep calling them “tiles”. Is “tiles” the right expression or not?
Charles Cipione: Tile, button. I’m not sure exactly how they referenced it internally.
Mr Beer: In any event, this one is displayed in the “Serve Customer” mode; is that right?
Charles Cipione: Yes.
Mr Beer: You were telling us that that was the mode that people might be most familiar with, SC, “serve customer”. Can you tell us about the other modes please?
Charles Cipione: So other modes include all of the back office processing that might need to be done, such as adjusting stock, REMing in or REMing out supplies and doing various housekeeping. So, essentially, there is the main mode or what I would describe as the main mode, which is the “Serve Customer” and a lot of other supporting modes to make sure that all of the housekeeping items or all of the ancillary work that needs to be done to manage the particular branch is available.
Mr Beer: When one is in a mode, the other tiles that are either displayed or available within that mode may increase or decrease; is that right?
Charles Cipione: That is correct.
Mr Beer: So the extent to which you can use the other buttons is affected by the mode that you have selected.
Charles Cipione: Yes.
Mr Beer: You mentioned in your answer there the REM-out supply division, ROSD, and REM-in supply division, RISD; can you explain the concept of REMing in and REMing out, please?
Charles Cipione: Yes. So the stock and cash at a particular branch needs to be – could be undersupplied or oversupplied, depending on the activity that’s occurred at that branch. If it’s undersupplied, the branch has the availability to reach out to POCL to provide more cash and/or stock as they receive that. That would be considered REMing in cash or stock. Alternatively, they might have collected too much, as far as cash or stock is concerned, at which point they want to return that to POCL or the Post Office, and that is REMing out.
Mr Beer: You describe in your report that there are three types of transactional data in the system: manual entries, transaction corrections and Fujitsu entries. Can you explain each of them, please?
Charles Cipione: Certainly. Manual entries would be the transactions that occur as the user is operating the Horizon counter. So as they’re conducting the business of stamps and other services available through Horizon, that is in the – that’s what is considered manual transactions and that would be most of the activity that one would expect to understand about the activity at that particular branch.
There are instances where some extra transactions need to be entered into the system, either to correct errors or to adjust for different situations and those two – one is called a transaction correction or a TC, and that is generated by the Post Office; and the second one would be when Fujitsu entered transactions to do the same type of corrective error. So one was sourced from the Post Office and one was sourced from Fujitsu.
Mr Beer: You mentioned lastly, in relation to this component, component A, of Horizon, that Legacy Horizon was coded in Visual Basic, C and C++.
Charles Cipione: Yes.
Mr Beer: It also utilised Oracle development tools and, as you have said already, the Riposte product. Where in your categorisation of the types of software do those species of software sit?
Charles Cipione: I’m sorry, could you repeat the question?
Mr Beer: Yes. You earlier categorised software into different types.
Charles Cipione: Oh, okay.
Mr Beer: Where in that categorisation do those three sit?
Charles Cipione: Yes, so Visual Basic, C and C++ are all programming languages. So I would include those in the application development category. The Oracle development tools, Oracle – these development tools would have been related to the relational database management system, the database that Oracle provides. So I would put that in DBMS. The Riposte product, that is an application – well – that’s a little bit of a mixed bag.
Mr Beer: A hybrid?
Charles Cipione: Yes. So the Riposte product is an application but it’s an application that was conformed to the Horizon product. So you could see there it was an application or an application development tool.
Mr Beer: Thank you. Can we turn to component C of Legacy Horizon, please, the messaging system. You say in the first line of paragraph 4.5.34 that the counter uses a messaging infrastructure provided by a system called Riposte. Can you explain what you mean, please, by a “messaging infrastructure”?
Charles Cipione: Yes. So a messaging infrastructure is a way of encapsulating data in a structured manner so it can be communicated in bits and bytes to other parts of the system. So it’s data – essentially it’s data with a lot of structure around it so that other components of the system can understand what that data means.
Mr Beer: You say that this was called Riposte, developed by Escher. You say:
“Everything that Riposte handles is stored as a message. Messages are constructed using a format known as Attribute Grammar.”
Then this sentence:
“This is a self-defining and nested record format that is technology independent.”
When I first read that, I started thinking about next summer holiday. Can you explain what you meant, please?
Charles Cipione: It’s a method. So the Attribute Grammar is a method of applying a structure to certain data – to data. So maybe an example might be – make it easier.
Mr Beer: Thank you.
Charles Cipione: So let’s say I wanted to send an email to someone, okay? Part of the components of that email would be who I wanted to send it to, part of the component would be the date I’m sending it, part of it might be the subject line and part of it might be the body of the email.
Using this messaging type structure, as I wrote that email and as it came off of my email system, it would go – as it was being transmitted there would be a tag that says, “This next bit of information represents who I’m sending it to and stop there”. Then there would be another tag that says, “This is the subject”, and then there would be another tag that says, “This is the date”, and then there would be another tag that says, “This defines the body of my email”.
All of that would be wrap inside perhaps some other tags that would help coalesce all that information, identify it as all being identified to each another and, as a package, that could be sent and then interpreted by another component of the system that would know – that would understand that structure and would be able to consume that information.
Mr Beer: You say they are a “tree-like structure” and could be considered a “proto-markup language”.
Charles Cipione: Yes.
Mr Beer: Can you explain what you mean by that, please?
Charles Cipione: So the example I just gave was anticipating that question, so a markup language is simply, number 1, defining that this is the beginning of my message and then it could have many elements in it. It could have one element in it. There would be indicators within that markup saying, this is – “You should expect five elements or you should expect two elements or you should expect elements until you see an end-of-element marker”.
But within each one of those markers would be the specific information that you’re trying to send and it’s identified very systematically and very structured. It’s easy to understand by a program but not necessarily easy to read by a human being. It’s an efficient way of categorising and organising information and delivering it for further consumption by other components.
Mr Beer: Can we see this in action with the hammer, screwdriver and pliers –
Charles Cipione: Yes.
Mr Beer: – and go on to 4.5.36 over the page. So, as you remind us, the purchases of two hammers, three screwdrivers, and one pair of pliers, using Attribute Grammar, you say the tree-like structure might use the following relationships, yes?
Charles Cipione: Exactly. Exactly, so –
Mr Beer: Can you talk us through 4.5.36, please?
Charles Cipione: Yes. So you’ll see that at the beginning of this structure I’m indicating that this is a sales transaction. So it’s marked there. Everyone that receives this will know that this is a sales transaction.
Mr Beer: Again, this is written in pseudo-code –
Charles Cipione: This is a pseudo-code.
Mr Beer: – for humans to read rather than the machine.
Charles Cipione: Yes, exactly, exactly. So underneath – so the first thing that this particular tree-like structure wants to communicate, in addition to that it’s a sale transaction, is who is the customer. So that’s the second element you see there. Then you’ll see that there is an indicator of how many items were purchased by this customer, just to give an indication. So there can be internal integrity checks to be done on what was built into this particular message.
Once we get past the count of the items purchased, we get to item 1, and this is more of the tree-like structure. Item 1 will have a quantity and a price and a total purchase amount related to item 1. Then we move on to item 2, and it will also have a quantity, a price, and a total purchase amount. That could go on infinitely or it could go on once, it really just depends on what the actual transaction is but the flexibility of this particular type of structure doesn’t require you to have just one or just five or just ten.
It allows – it’s a flexible way of storing information and, at the end of this all, you see that there’s the total basket purchase amount.
Mr Beer: Looking at subparagraph 37 there, you translate that into our example of two hammers, three screwdrivers and one pair of pliers.
Charles Cipione: Yes, exactly.
Mr Beer: Can you just talk us through that? It may be obvious, given what you’ve just said.
Charles Cipione: So we see that the sales transaction, there’s a start and an end marker on the sales transaction. You can see that that is the giant wrapper around this particular message. The first element within that particular wrapper is the customer name, which I chose “John Doe” as the customer and, past that, you’ll see that this message contains three items that were purchased.
The first purchased item name was a hammer with a quantity of two and a price of £5 which totals to £10 for that particular item purchase. The second item is a screwdriver, quantity of three, with a price of £7, which results in a total purchase amount of £21. The third item is pliers with a quantity of one, a purchase price of £6 and a total purchase amount for item 3 of £6. Then it sums up all of the purchase amounts as £37 and ends.
Mr Beer: Thank you. Can we go forward to page 166 of your report, please, which is Appendix B. 166, thank you. Can we see here an example of Attribute Grammar actually written in code?
Charles Cipione: Yes.
Mr Beer: This isn’t, I think, the hammer, pliers or screwdriver example?
Charles Cipione: It is not. Like I indicated earlier, this is not human friendly to read. You would have to understand the definition of the structure, see what elements were available within the structure to properly interpret this. But anyone using this particular Attribute – if this Attribute Grammar was the standard for this particular system, every component of the system should be able to interpret this properly.
Mr Beer: Would the coder be able to interpret it?
Charles Cipione: It depends on what coder we’re talking about. Not all current coders would need to be able to interpret this because there could be a group of coders that are designated simply to do the translation of this information. So there might be one group of like four or five coders whose complete responsibility is to make sure that this structure is understandable to all systems and to write the code that can both encode and decode information passed in this format.
Mr Beer: Lucky them!
Charles Cipione: Yes.
Mr Beer: Can we go back, please, to page 39 of your report and paragraph 5.39 at the foot of the page. You say:
“Normal transactions at the Counter take place within a customer session. Each physical transaction with the customer … results in the creation of one or more messages depending on the complexity of the transaction.”
Then you say this:
“For example, a stamp sale has one message, and a postal order results in two messages (one for the postal order and one for the fee). None of these messages is normally written to the message store until the customer ‘settles’ the session.”
Can you just explain this, please?
Charles Cipione: Yes, so what this is indicating is that, as the clerk is transacting the business on the counter and adding in different items to the transaction, it’s not recorded until – there’s no need to record this information until the transaction has become official by settling it, by receiving some form of payment, so that’s what I’m trying to communicate here.
Mr Beer: So the stamp results in one message but the postal order results in two messages. Can you just explain that?
Charles Cipione: Certainly. So for some transactions there might just be one item that needs to be recorded in a message and for other – so the stamp. That’s a fairly simple, straightforward action. I want to pay £5 for a set of stamps. In the Horizon System, that’s just one transaction. It’s a simple – I’ve sold this book of stamps for £5.
However, for accounting purposes or managerial purposes, other types of transactions might need to segregate, you know, for instance, the postal order. A postal order might have two components where one is the amount of the postal order and then one is the amount for the fee that is being charged to issue the postal order. So, in that instance, the Horizon System is recording two messages.
Mr Beer: You say that a key feature of each session is that they are zero sum, that’s to say that the debits and credits of the transactions must sum to zero; is that right?
Charles Cipione: Yes.
Mr Beer: Over the page, please, to 4.5.40. On completion of a session by a clerk then the resulting transactions are saved locally on the counter, you say, in something called the “counter message store”, also known as the “Riposte message store”; is that right?
Charles Cipione: Yes.
Mr Beer: After that sentence, you say:
“Where there is more than one Counter in an Outlet, the Riposte Message Store is replicated across all the Counters. Where there is only one Counter, the Counter contains two mirrored disks, one fixed and one exchangeable, so that the message store can be recreated on a replacement Counter if necessary …”
Can you just explain that, please?
Charles Cipione: Yes, so this harkens back to the issue we were talking about very early in the discussion this morning about the difficulties of systems in the time period we’re talking about. One of them is making sure that there’s the proper amount of redundancy of data to the extent that there’s a hardware failure. The way the Horizon System handled the possibility of hardware failure – when I say “hardware failure”, I really mean that maybe a hard drive goes bad, because all of this information that is being stored locally is being stored on a hard drive.
In a single-counter branch – well, let’s talk about – in a multi-counter branch all of the counters can communicate with each other through their hub or their router and that allows the redundancy to be across different counters. So as a message store on one counter is created, Horizon creates a redundancy of that message store on the other counters, in case of hardware failure on the original counter.
For a single-counter branch that multi-counter redundancy is not available to them. So for a single-counter branch, the way that they account for – the way they make sure that there is a redundancy availability is to have two hard drives within the counter mirroring each other, so that way if one hard drive fails, they always can revert to the second hard drive to recover the information.
Mr Beer: You explain in subparagraph 41 the data replication facility that Riposte had. Can you explain that to us, please?
Charles Cipione: So this is more aligned with the fact that the communication system might not be online at the time that the transactions or the messages are being stored. So remember that the transactions, you can think of that as a real world term, but the Horizon digital equivalent to that is a message. So to the extent that there is no communication between the counter and the campuses, the Horizon System, at least, you know, in the late ’90s allowed for an accumulation of all of these messages, if they knew that they were in an offline position, to continue to accumulate those until the connection was made again.
Then the replication facility would work through all of those stored messages, send them back to the campus. Once it got confirmation from the campus that all of those were restored, it finished the process. But that whole check and balance was just to make sure that the campuses received all of the messages that were stored at the counter while it was in an offline position once it came back online.
Mr Beer: Thank you. In that subparagraph 41, you say that the counter’s messages would be synchronised with a version of the message store saved on the Legacy Horizon campuses in their correspondence servers. Can you just explain what the correspondent servers were?
Charles Cipione: Certainly. The correspondence servers were the communication facility at the campuses that basically allowed for the data traffic to move between the counters and the rest of the operations at the data facility or at the campus.
Mr Beer: Okay, we’ll come back to those in a bit more detail when we get to component D.
Charles Cipione: Yes.
Mr Beer: Lastly, in subparagraphs 42 and 43, you explain reference data, please. Can you explain to the Chair the three categories of reference data that you mention there?
Charles Cipione: Yes. So the reference data – I broke up the reference data into the three categories. So the reference data could have been derived from POCL or the Post Office. This might be – like, for example the hammer, screwdriver, plier list, to the extent that the POCL had a list of prices for all of their products, their stamps and what not, that would be an example of reference data coming from POCL.
There was also similar information coming from POCL’s clients, you know, price lists and other information that the client transactions relied on to process correctly.
Mr Beer: So that’s the DWP or somebody like that?
Charles Cipione: Exactly, exactly.
Finally, there’s reference data that was more behind the scenes reference data. This is reference data that the Horizon System used internally to make sure that the Horizon System was operating efficiently. So this would be the – this would be a more complicated example of a dataset – a data-driven logic. The Horizon design had a lot of data-driven logic for, for instance, how the menus showed up. What menus were available in the different modes on the user interface on the counter. That would be an example of some information that was actually kept in some reference data and not necessarily coded in the source code.
Mr Beer: All of those three categories of reference data, they were saved locally on the counter in the counter message store; is that right?
Charles Cipione: Yes.
Mr Beer: That localised copy, is this right, is one of the features that enabled trading to carry on, an operation to carry on, even if connection with the campuses was lost?
Charles Cipione: Yes.
Mr Beer: Can we turn, then, to category or component D, please, the Horizon campuses. Can we look over the page at page 42 to the table at 4.10, please, and just highlight or blow that up, please. Thank you.
Could you talk us through the Legacy Horizon campuses, the layers, from the bottom upwards, please, starting with the counter layer –
Charles Cipione: Yes.
Mr Beer: – which you’ve already dealt with.
Charles Cipione: Yes. So we just talked about the counter layer but, essentially, all of the information is stored in messages. All the transactions are stored in messages at the counter layer, and that’s generally what’s being communicated to the campuses, and that’s being facilitated by what is labelled here the “Correspondence layer”. That layer is – that is the layer responsible for receiving the information into the campuses as well as pushing the information from the campuses back down to the counters.
Mr Beer: So that’s essentially the next layer up. The correspondence layer is the communications of the network; is that right?
Charles Cipione: It’s the piece of the – it’s the piece of the campus that communicates with the counters. Yes.
Mr Beer: You say in your report that that layer and the counter layer, shared the use of the Riposte message service, RMS, which was a message storage and replication mechanism running on both the correspondence servers and on the counter; is that right?
Charles Cipione: Yes, that’s right.
Mr Beer: You say that supported a shared distributed message store that ensured that information penetrated at the counter is replicated at the campuses and vice versa –
Charles Cipione: Yes.
Mr Beer: – and that those Riposte mechanisms interact directly with the agent layer, a specialised Riposte Archiver running on the correspondence servers was used to ensure that all Riposte messages are written to tape for audit purposes. Can you just explain that, please?
Charles Cipione: Yes. So the agent layer, I would say that that would be your universal translator for the use of all the information, both operationally and financially, once it got through the correspondence layers. So one of the items like you said is – you know, it made the information available to be written to archive or to tape. It also translated it into the multiple different consumers of this information that we’ll get to in a second, but it was where the translation from the Riposte message store format to whatever format was needed by any one of these connected services appeared, and it will also be able to reverse that translation if one of these other services needed to push information back down to the counter. The agent layer would be the layer that translation occurred at.
Mr Beer: You say it provides facilities to pass messages directly to third-party clients and to return the client response back to the counter – is that right –
Charles Cipione: Yes.
Mr Beer: – and that this layer, the host layer, applies any business rules to the information being received from or sent to the external client?
Charles Cipione: Yes.
Mr Beer: Can you deal with next on the table the RDMS, the data reference management system?
Charles Cipione: Yes. We spoke about this a few minutes ago and often throughout the discussion today, the Horizon System relied on reference data to operate correctly and, if we remember back to my theory section, it’s very important that reference – you know, when a system does rely on data-driven logic, it’s important that there be a system that ensures the integrity of that data within the data-driven logic. That’s – the responsibility of the RDMS system is to do the maintenance updating and control the integrity of the information that’s being used as reference data within the Horizon System.
Mr Beer: So, as you say, its integrity – the integrity of the RDMS, is important to the proper operation of Legacy Horizon?
Charles Cipione: Yes.
Mr Beer: Can you next deal with the data warehouse, please.
Charles Cipione: Certainly. The data warehouse is a place where – it is a facility where information is collected for further analysis by POCL. There are a number of different analyses or reporting requirements that POCL had. This data warehouse is basically the data management layer that responds to providing information for reporting and analytics.
Mr Beer: You tell us in your report about the transaction processing system or TPS, which is the next box to the left, at this layer. Can you tell us what function that had, the TPS?
Charles Cipione: Yes. So all of the transactions that are collected by Horizon ultimately terminate at POCL – in POCL’s financial systems. The TPS system is what translates the transactions that are collected from Horizon and delivers them to the POCL’s TIP system, which is their transaction information processing system, which is what – it’s POCL’s internal system for their accounting processing.
So, basically, it delivers all of the managerial information, all the managerial accounting system to POCL so they can use it for their financial reporting as well as their managerial aspects of their business.
Mr Beer: You say that that system, TIP, the final POCL system, was one used by POCL to collect transaction records about all transactions that occurred at branches –
Charles Cipione: Yes.
Mr Beer: – that such transactions were gathered, in the first instance, by the transaction processing system once the counter had set the end of day marker. Can you tell us what the end of day marker is?
Charles Cipione: That’s an indicator that says where – “This is a good set of transactions for this day and we’re ready to deliver them to be processed further by POCL” or whatever else needs to process these transactions.
Mr Beer: You say in your report that Legacy Horizon was not an end-to-end accounting system but the data that it passes to Post Office Counters Limited and its clients had to be sufficient to enable them to balance their own books and to settle accounts between them?
Charles Cipione: Yes.
Mr Beer: I think I have not included, so far, the management information services MIS. Can you explain what that is, please?
Charles Cipione: Oh, yes. So this was an adjunct to the data warehouse, to continue to further assist POCL in analysing the data. But this was, I believe, specifically used to detect errors or had been an error detection system for the transactions that were received from the Horizon System.
Mr Beer: Lastly, it may be obvious, external interfaces in the top right, Post Office Counters Limited and clients. Just explain what that’s meant to be?
Charles Cipione: Yes. So, as you know, there are instances that the Horizon System needed to reach out and speak directly or communicate directly with the external client community. That’s what this represents. You know, to the extent that network banking services needed to get a verification of sufficient funds to do a withdrawal, that would be represented here.
Mr Beer: Thank you very much. I think that’s the end of component B and, therefore, the end of the description of Legacy Horizon’s three main components?
You speak in the next paragraph of your report, 4.5.53, which is over the page, about software updates made to Legacy Horizon between rollout in 1999/2000 to the introduction of Horizon Online in 2010. You highlight seven or eight of the most significant of them. Can you just talk us through those? Firstly, the Core Systems Release that was introduced in August 2000.
Charles Cipione: Yes, the Core Systems Release introduced the Automated Payment Services, which supported payments by customers to the utility companies, such as British Telecom, electricity and other utility companies, as well as other clients using barcoded bills, magnetic cards or smart cards. It also introduced reconciliations between the Automated Payment Service and the data harvested by the APS agents and data harvested by TPS. So a lot of internal reconciliations functionality appears to be added here.
I also note that it seems that there was also Core Systems Release Plus released, which I would assume was closely aligned with Core Systems Release.
Mr Beer: In February 2000, there was an upgrade called Maintenance Release M1 and you say that the main purpose of that was to enhance the CSR+ applications.
Charles Cipione: Yes, yes, that’s correct.
Mr Beer: In June 2001 there was a release called S06 Release Day D Rectifications; is that right?
Charles Cipione: Yes.
Mr Beer: That included a fix. We’re not in this table or this paragraph looking at all of the fixes that were released; is that right?
Charles Cipione: That is correct.
Mr Beer: In 2002/3, Network Banking Service, NBS, Debit Card Services, DCS, and Data Reconciliation Services were introduced.
Charles Cipione: Yes.
Mr Beer: You describe in your report what reach of those three features are?
Charles Cipione: Certainly, sir. The Network Banking Services allowed customers of selected banks and building totals withdraw money from or deposit money into their bank accounts. So it allowed a banking option functionality to be accessed through the Horizon System.
The Debit Card Service was – allowed customers to have had the functionality of using their debit cards to pay for goods through the Horizon System.
The Data Reconciliation Services related to all the NBS and DCS transactions and helped reconcile them for the different data flows. So it was an internal reconciliation facility.
Mr Beer: In subparagraph (f) you mentioned a development in 2004 when Post Office Limited Finance System was implemented, and you described that as an SAP accounting system. What do you mean by that?
Charles Cipione: SAP is – in fact we used this as an example earlier, SAP provides ERP systems, a part of which are the accounting. It provides an accounting application and it looks like the Post Office bought and installed that – implemented that in 2004.
Mr Beer: What do you understand this release to have done?
Charles Cipione: This essentially took the TIP communication out of the loop from – so TPS used to deliver their transaction information to TIP. Now it delivered that transaction information directly to SAP.
Mr Beer: Thank you very much. You say that in paragraph (g), over the page, that at some point in 2004, a program called IMPACT was delivered and this included a series of updates. I think there are six of them that you mentioned; can you just talk us through those, please?
Charles Cipione: Yes. So the – a few of them were changing the processes at the branch. The first one I cite here is there were rollovers, which we’ll talk in a bit, but rollovers used to be every week, now they were going to be extended to a four to five-week period. Non-value stock declarations weren’t required and weren’t required by the stock balancing process any more.
A concept of a local suspense account was introduced for the processing of variances. Stock units were no longer allowed to carry discrepancies over and any – over cash accounting – over trading periods, and they had – and any discrepancies that arose out of the stock unit balancing were sent into a local suspense account when the stock unit rolled over.
Additional checks were carried out in order for the final stock unit to roll over, which included that the stock unit wasn’t allowed to roll over if there were any outstanding transaction corrections, and local suspense accounts had to be settled before the final stock unit could be rolled over to the next traded period.
Then, finally, there were changes to the data server that were made to reduce the number of times that the message store was scanned to pick up transactions during the balancing process. A Riposte mechanism known as Notifications was added – was used to add new transactions to existing totals as further transactions were generated, so this was more of an operational, behind-the-scenes update.
Mr Beer: Thank you, Mr Cipione. Can we turn over the page, please, to page 46 and undertake exactly the same exercise, looking at components (a), (c) and (d), missing out component (b), but in relation to Horizon Online, rather than Legacy Horizon. You say that you understand that the online system, HNG-X, as it was then known, was piloted in late 2009 and rolled out in the spring of 2010.
Charles Cipione: Yes.
Mr Beer: You kindly outlined the reasons for the release of Horizon Online, as you understand them from the documents, to be first reducing the cost of running Horizon. Can you explain that in a little more detail, please?
Charles Cipione: Yes. The amount – I would imagine that the communications cost or the old – the inefficient communications infrastructure and all of the related hardware redundancy that was required to support that old technology was rather costly. With the online system, there wasn’t a need for much of that redundancy, and that was one of the factors that helped reduce cost in the Horizon Online system, as opposed to the Horizon Legacy system – or Legacy Horizon system.
Mr Beer: Secondly, you understand from the documents, the second reason to make the change was in order to take advantage of improved communication technology reliability; is that right?
Charles Cipione: That is correct.
Mr Beer: You say that that change meant it was feasible to have branches that were online for the vast majority of time. That benefitted a wider change in the business as an increased proportion of transactions involved NBS and DCS. Can you remind us what they were?
Charles Cipione: Yes, sir, the banking system and the debit card system. Those were the two bits of functionality that were required – that required Legacy Horizon to be in online mode to operate correctly. With Horizon Online, it was assumed that the communication facility would be persistent indefinitely and it allowed for that facility to be available all the time – or those facilities.
Mr Beer: Thank you. The third driver that you understand to have operated, from the documents, was in order to simplify the design of the user interface. Can you explain what you understand by that?
Charles Cipione: Yes. So the – I’m sure, as the original Legacy Horizon was rolled out, that there were probably a lot of comments on the usability of the system. It looks like they took the opportunity to simplify the number of screens that were required to manoeuvre through to do the same functions that the Legacy Horizon system did, in many more steps.
Mr Beer: Then, lastly, in order to simplify business processes. Can you explain that, please?
Charles Cipione: Yes, so, you know, in addition to just simplifying the screens, that implies a simplification of the processes behind the screens, but it also includes, you know, items such as reducing the number of reports that were available to the SPMs at the branches.
Mr Beer: With that introduction in mind, can we turn to component A, which is, again, the counter and the peripherals. Was it right that the hardware components for Horizon Online were almost identical to those for Legacy Horizon?
Charles Cipione: Yes.
Mr Beer: However, an important change was that every branch received a new router, a piece of hardware which allowed the branch to connect to the data centre?
Charles Cipione: Yes.
Mr Beer: Did that router have a physical line connecting it to the data centre?
Charles Cipione: Yes.
Mr Beer: But –
Charles Cipione: It could have been, possibly – it was either a fixed or a mobile line but it was, effectively, a persistent communication link to the data centre.
Mr Beer: You describe in your report that the new architecture under Horizon Online had four layers: presentation, interaction, business and services. Could we look at those, please, in 4.6.5 of your report?
Charles Cipione: Yes.
Mr Beer: Can you explain what the presentation layer was?
Charles Cipione: So the presentation layer is simply the way the user sees the screen and how the user interacts with the system.
Mr Beer: The interaction layer?
Charles Cipione: That was the underpinnings of the presentation layer, for instance, you know, how the menus were organised, and it was also – it also replaced – so now that we’re online and now that – and we’ll get to this in a second – we don’t need the Riposte messaging technology in exactly – we don’t need that exact mechanism to deliver the data now. It also will help to replace how the transactions were communicated to the campuses.
Mr Beer: Thirdly, you say that the business layer provided the business applications in an object-oriented manner. What do you mean by that, an “object-oriented manner”?
Charles Cipione: So “object-oriented manner” is a programming term and it has to do with more efficiently organising your code for repeatability and it’s a style of programming. I think that would be sufficient for explanation right now.
Mr Beer: Then, lastly, the layer – the services layer. Can you explain what that was, please?
Charles Cipione: Yes. This is – so the service layer is, you know, the set of objects that support all the applications and the processing engine. It basically set the sequence of processing steps for the counter to deliver its services.
Mr Beer: Under Horizon Online, is it right that customer transactions were not stored at the counter level?
Charles Cipione: That’s correct.
Mr Beer: At the counter level, data including reference data, process definitions and report definitions were stored at the counter?
Charles Cipione: Yes.
Mr Beer: You say the service layer is the only layer of communication with the data centre. Can you explain that, please?
Charles Cipione: Um, it’s – the service layer, that was – all the functionality for communication resides in the service layer and that’s simply what I’m trying to communicate there.
Mr Beer: The figure at the top of this page, at 4.11, you depict the counter setup in a multi-counter branch in Horizon Online, is that right?
Charles Cipione: Yes.
Mr Beer: Can you just talk us through that please?
Charles Cipione: Certainly. So if you remember from a multi-counter example from the Legacy Horizon, the configuration required one of the counters to be the Gateway PC, which was responsible for communication directly to the data centres or to the campuses. All of the other counters basically were connected to that Gateway PC. The difference here is that the counter, each one of the counters are connected through the communication – the router hardware, which directs the transmission of the transactions to the campuses.
So, really, what this is showing is that each one of the counters can operate independent of the other counters and, as we talked about, there is no requirement or, in fact, it’s not allowed for the messages – the transactions to be stored in messages on the counters. The messages are communicated directly to the campuses because it’s a persistent online connection and there’s no need to worry about having them persist at the counter level.
Mr Beer: Thank you. You say that the biggest change for a subpostmaster would have been to the user interface which changed significantly, and I think we can see that over the page in figure 4.12 at the top of the page. If we can just highlight those two diagrams next to each other, those two screen prints. At the top of the page, please. Thank you.
Charles Cipione: So you can see the look and feel is quite different and, as I described a few minutes ago, the actual procedure through the screens also was different based off – you know, produced a more efficient set of business processes that were represented by a different set of screens.
Mr Beer: Additionally, I think you tell us that there were some changes to the available functionality, for example the ability to transfer sessions between counters was removed when Horizon Online was introduced; is that right?
Charles Cipione: That’s correct.
Mr Beer: That table can come down. Thank you. Horizon Online was coded using java. That’s different, isn’t it, because it replaced the Visual Basic components that had been used under regular Horizon?
Charles Cipione: Yes.
Mr Beer: Skipping over component B, then, and going straight to component C. You tell us that because Horizon Online only stored transaction data at the data centre and not at the counter, as you’ve explained already, the Riposte message centre was no longer required?
Charles Cipione: Yes.
Mr Beer: Reference data is, however, still being stored locally on the counter; is that right?
Charles Cipione: Yes.
Mr Beer: There was a change in messaging system using XML instead of Attribute Grammar –
Charles Cipione: Yes.
Mr Beer: – is that right? Do you know why that was; can you tell from the documentation?
Charles Cipione: It was – I am going to speculate but I’m going to say that the XML and Attribute Grammar were different versions of the same concept and XML was probably more universally accepted method of markup language as opposed to the Attribute Grammar and they just took this opportunity to switch. But I don’t know that for sure, but that’s what I suspect.
Mr Beer: Then, lastly, you say a difference is that Horizon Online used the TCP/IP protocol instead of UDP/IP to send data between the counter and the data centre. Can you explain that, please?
Charles Cipione: Yes, it’s just more of an updated communication protocol that they used.
Mr Beer: Then, lastly, in terms of the component parts of Horizon Online, component D. You say there were various updates made to the data centre. You highlight two: the branch access layer and the branch database being new elements introduced in Horizon Online. Can you just explain each of those, please?
Charles Cipione: Certainly. So since the – there was a switch in how the transactions were stored and communicated to the data centres, meaning that there’s now a direct communication of the transactions to the data centre, the branch access layer was responsible for facilitating that communication and the branch database was essentially the management of that information after the branch access layer received the information at the campus.
Mr Beer: Thank you.
There are a number of updates to the Horizon Online system. For our purposes, the most significant was in 2017 – is this right – as you’ve mentioned already: the release of HNG-A Horizon Online Anywhere?
Charles Cipione: Yes.
Mr Beer: You say that you haven’t been provided with any substantial documents which detailed the changes delivered, however you understand that one of the major changes was that the operating system was upgraded from Windows NT to Windows 10. That was necessary because of the obsolescence of Windows NT?
Charles Cipione: Yes.
Mr Beer: Thank you.
Sir, I wonder whether we could take the afternoon break there and come back at maybe 3.10?
Sir Wyn Williams: Certainly.
Mr Beer: If you’re willing to sit a little later than usual today, sir, we could finish Mr Cipione’s evidence today.
Sir Wyn Williams: Well, I’m perfectly happy to do whatever is necessary to accommodate Mr Cipione and I’m sure he’d much prefer to finish than come back tomorrow.
Mr Beer: Thank you very much, sir.
(2.56 pm)
(A short break)
(3.10 pm)
Mr Beer: Good afternoon again, sir. Can you see and hear me.
Sir Wyn Williams: Yes, I can, thank you.
Mr Beer: Thank you very much, sir.
Can we turn, Mr Cipione, to balancing and rolling over. You tell us in paragraph 4.7.1 of your report about a process on Horizon which was prominent, which you see referenced in the PinICLs, PEAKs and KELs, and that is the concept of a cash accounting period. Can you just tell us what a cash accounting period is, please?
Charles Cipione: Yes, the cash accounting period was a weekly cycle – I believe it started Thursday and ended on the subsequent Wednesday evening – that was basically a period of time at which point the – a reconciliation needed to be done. All the books and records needed to be checked to make sure they were in alignment with the Horizon System to the physical stock and cash that was at the branch.
Mr Beer: You say that in 2004 you understand that Post Office Counters Limited moved to a monthly trading period, a TP cycle, with months made up of four or five weeks; is that right?
Charles Cipione: Yes.
Mr Beer: The cash accounting periods, the CAPs, were sequentially numbered one, two, three, four, five, and mirrored the financial year, starting in March each year and running to 52 or 53 weeks later?
Charles Cipione: Yes.
Mr Beer: You say that the cash accounting period is of particular interest as it acted, as you’ve just said, as a weekly reconciliation point for a branch?
Charles Cipione: Yes.
Mr Beer: You say that the data stored in Horizon was compared with the cash and stock physically held in the branch at the end of the cash accounting period and that this weekly reconciliation process was referred to as “balancing”?
Charles Cipione: Yes.
Mr Beer: You say that this was undertaken for each stock unit in the branch?
Charles Cipione: Yes.
Mr Beer: Only once subpostmasters had balanced all of their stock units were they permitted to roll over the branch until the next cash accounting period?
Charles Cipione: Yes.
Mr Beer: That process of balancing the stock units and moving to the next cash accounting period were commonly referred to as “rolling over” or a “rollover”.
Charles Cipione: Yes.
Mr Beer: You explained that a cash accounting period is further subdivided into balance periods. Can you explain those, please?
Charles Cipione: Yes, a balance period was – could be shorter than the cash accounting period. It didn’t have to be but it could have been shorter. It was just a mechanism that allowed the SPM, the subpostmasters and subpostmistresses to balance incrementally on an interim basis before the end of the cash accounting period.
Mr Beer: You tell us that the cash accounting period and the balancing period were each displayed on the Legacy Horizon screen and the trading period and the balancing period were displayed on the Horizon Online screen?
Charles Cipione: Yes.
Mr Beer: Can we just look at those, please, go back to page 48. Look at page 48, please, and look at the table at the top of 48, the pair, side by side. If you can highlight those. Thank you.
So you tell us that the cash accounting period and balancing period were displayed on the left-hand screen, which is the Legacy Horizon screen. Could you point out where they are on there?
Charles Cipione: Yes, I’m afraid that we must have switched out the graphic on this – on the Legacy Horizon system because it doesn’t show that particular information but, on the Horizon Online, you can see the balance period and the trading period on the bottom. But it was inadvertently removed from this particular graphic.
Mr Beer: Do you see on the left-hand screenshot –
Charles Cipione: Oh, I’m sorry.
Mr Beer: – to the right of the line, two lines from the bottom in white it says, “TP:07/BP:01”, I think.
Charles Cipione: Yes, that is correct, I apologise, I think my reference in the report was back to a prior graph, in which case it wasn’t there, so I apologise.
Mr Beer: Exactly. That’s why I went to this one rather than the one you referenced in your report because I think when you were screenshotting, that got cut –
Charles Cipione: Yes.
Mr Beer: – in the figure on 4.8 on page 37 but I think it is actually shown on this one?
Charles Cipione: It is.
Mr Beer: Then on the right-hand, on Horizon Online, can you point out where the TP and BP are shown on that, please?
Charles Cipione: On the bottom bar you can, right as the white shows up above it, the “TP:01” and the “BP:03”, if my eyes are working correctly.
Mr Beer: Can I ask, is there a facility to use a pointer on this display system.
Can you point, please, Frankie, to the TP and BP on the line at the foot on the right-hand diagram. Perfect. Then the BP? Thank you very much.
I hope that’s visible to you, sir.
Sir Wyn Williams: Yes, it is, and could we just do it with the Legacy Horizon as well, just to be – so that I’m sure I’m looking at the right thing.
Mr Beer: Left-hand diagram, thank you.
Sir Wyn Williams: That’s it. I’ve got it. That’s fine.
Mr Beer: Thank you very much.
We can go back to page 50 of the report and we’re up to paragraph 4.7.5, where you say that, in order to balance and roll over, a subpostmaster had to undertake various steps, a summary of which you reproduce. This process would be undertaken for each stock unit that the branch operated and you outlined five processes. Can you just walk us through those, please?
Charles Cipione: So first, (a) check all the stocks held in branch against the system held values and adjust the values in the system where, so to the extent that there was a difference, enter what was actually physically on hand; declare stamps held in the branch; declare the cash held in the branch; then produce a balance sheet – a balance snapshot report and complete the mandatory checks, making adjustments to transactions and stock and cash declarations where inconsistencies are identified, or accepting any discrepancies that the Legacy Horizon system identified between its calculated values and those from the declarations; and then confirm in Horizon that they wished to roll over this particular stock unit to the next CAP.
Mr Beer: Thank you very much. You say in 4.7.6 that:
“Any loss or gain that was identified through this process must be either posted to the Suspense Account … or had to be corrected by the [subpostmaster] adding funds to the till (if a loss) or [taking money out] (if a gain).”
Charles Cipione: Yes.
Mr Beer: You explain that the posting of discrepancies to the suspense account was only made once the stock unit had rolled over to the next cash accounting period. Can you just explain that?
Charles Cipione: So as the – as it’s rolled over, so the discrepancy as that was rolled over to the next counter unit would then cause the transaction – basically cause an update to the suspense account.
Mr Beer: Thank you.
You explain that, once stock units had been balanced and rolled over, a subpostmaster would produce the cash account report for the branch, which would summarise the position across all stock units and then the subpostmaster would check this report and, if they were happy with this, they would roll over the branch to the next cash accounting period; is that right?
Charles Cipione: Yes.
Mr Beer: You explain in 4.7.9 that that which you have described is intended to provide an overview of what steps were followed as part of the rollover and provides the context for the process of checking that receipts and payments matched.
Charles Cipione: Yes.
Mr Beer: You say that the stock unit balancing process consists of accumulating all of the receipts for the stock unit concerned and all of the payments for that stock unit in the period for which the report is being produced and ensuring that the total value of receipts matches the total value of any payments. When that state is reached the stock unit is said to have balanced, yes?
Charles Cipione: Yes.
Mr Beer: You explained some definitions of payments and receipts, which might be obvious but can you just talk us through those, please, in the next paragraph?
Charles Cipione: Certainly. Payment is essentially a transaction that was a payment to the customer in and a receipt is a transaction that’s a payment from the customer.
Mr Beer: Thank you.
You explain that it’s not intuitive that payments and receipts should match one another, however it’s your understanding that the balancing of payments and receipts factored in the cash and stock balance at the end (sic) of a cash accounting period, as well as the cash and stock balance at the end of a cash accounting period; is that right?
Charles Cipione: Yes, it takes into account the beginning balances and the ending balances.
Mr Beer: You’ve set out – and we needn’t go through them – at 4.7.14, the balance equations for a stock unit in six examples.
Charles Cipione: Yes.
Mr Beer: You draw these threads together at paragraph 4.7.15 on page 53, in an example table at table 4.4. Can you talk us through this, please?
Charles Cipione: Certainly. This is a simple example. We start off with cash – a brought forward balance for both cash and stock, which, in this case, just represents stamps. The cash was £5,000 and the stock was £500. You can see that the different receipt amounts relate to payment for a TV Licence, £100; payment of road tax for £75; Alliance & Leicester giro deposit, £150; the purchase of First Class stamps for cash of £5; and additional money that was REMed in from POCL of £100.
On the other side of the ledger, you see A&L giro withdrawals of £50; a pension payment of £25; savings withdrawals of £100; and the issuance of the stamp of £5 to a customer, which results in the carried forward balance of cash, £5,255, and stock, which is £495.
What this report shows, or the way this report should be read, is you can see that the – everything on the left side, which is the total receipts, adds up to £5,930 and everything on the right, for the payments, also adds up to £5,930. You’ll notice that the only stock transaction was the £5 of stamps for cash and you’ll notice that that is – that the stock is being reduced by £5 and the cash is being increased by £5 for that particular transaction. So that’s the only one that really goes between the stock and the cash.
Mr Beer: So for this cash accounting period, from 14 through 15 and then into 16, the total for receipts, 5,930, matches that for payments, 5,930. So the stock unit is balanced and therefore could be rolled over without the need for adjustment or any further action?
Charles Cipione: Yes.
Mr Beer: I think it’s fair to say that this side-by-side analysis here is not something that you were able to see when you looked at the data that you were provided by the Inquiry?
Charles Cipione: That is correct.
Mr Beer: Is that because the format of a stock unit balance report that was produced by Horizon Online, the manner in which it produced its stock reports, was different, because the way it came out of the tally printer was a sequential list rather than a more useful table like this?
Charles Cipione: Yes.
Mr Beer: Thank you very much. Can we turn to our penultimate topic, please: error logging and remediation on Horizon. This starts at page 55 of your report. Thank you.
You say that as part of the incident management process for Legacy Horizon, ICL utilised a Fujitsu proprietary core management system for logging errors and defects. What do you mean by a proprietary core management system?
Charles Cipione: It was built by and for Fujitsu.
Mr Beer: This was known as PinICL?
Charles Cipione: Yes.
Mr Beer: You say it was created from another ICL programme with changes made at the request of ICL Pathway; is that right?
Charles Cipione: Yes.
Mr Beer: It was used between ‘96 and 2003, prior to the PEAK system – about which we’re going to hear in a moment – being introduced; is that right?
Charles Cipione: Yes.
Mr Beer: You explain at 5.1.2 the various ways in which a PinICL can be raised. Can you just explain those to us, please?
Charles Cipione: Certainly. PinICL could have been raised through internal testing or monitoring. So, you know, let’s go back to our testing process. You know, the quality assurance could have raised a PinICL or, you know, the development team could have raised a PinICL. It also could have been raised from feedback from the user community, either through the user acceptance testing phase or from the deployment and, you know, maintenance phase of the work – of the Horizon cycle.
Mr Beer: Post Office Counters could raise incidents themselves –
Charles Cipione: Yes, that is correct.
Mr Beer: – ie if a subpostmaster fed incidents back to them, they could raise a PinICL with Fujitsu?
Charles Cipione: Yes.
Mr Beer: You say, I think, lastly, that through the Horizon System Helpdesk –
Charles Cipione: Yes, so users – users of the system would not have direct access to the PinICL system. Rather, they would have access to the helpdesk facility that was put together by ICL Pathway.
Mr Beer: That was called PowerHelp, yes?
Charles Cipione: Yes, that is correct. The internal team – so while the end user wouldn’t directly raise a PinICL, if it was deemed appropriate by their level 1 and level 2 helpdesk people at ICL Pathway, it could eventually become an item in the PinICL system.
Mr Beer: You helpfully describe for us the lines of support that were available, the first of which was the Horizon System Helpdesk, HSH, which you describe as the first line of support being responsible for recording the details of an incident diagnosing the problem and attempting to resolve an issue. If it was unable to do so, it would be routed to a second level of support called the System Management Centre or SMC; is that right?
Charles Cipione: Yes.
Mr Beer: You say that the SMC would determine if the incident was a software code problem. How would it do that?
Charles Cipione: Um, the SMC would invest – you know, or would read the description, make a determination based off their training and that was, basically – that was their job: to determine whether it was a software problem or not.
Mr Beer: You say that if the problem was a known error, ie something described in a KEL, a Known Error Log – is that right –
Charles Cipione: Yes.
Mr Beer: – they would see whether there was a work around that was written into the KEL and presumably suggest that workaround?
Charles Cipione: Yes, that is correct.
Mr Beer: If there wasn’t a workaround, the System Management Centre would ensure that there was no duplicate calls for the same problem. Just explain that?
Charles Cipione: Right. So it’s very possible that if there were an issue with Horizon, there could be multiple calls coming in about the same issue, and perhaps there was a difference in timing when, you know, call 1 to call 10 came in. To the extent that a PinICL had already been raised for a particular issue, there was no reason to raise a second PinICL for that same issue. So they would simply acknowledge that this call was related to a PinICL – to an issue that had already been raised, and not raise a second issue.
Mr Beer: If there was no known error log or no known workaround in a KEL, if there was no duplicate call or if this call was attached to a duplicate call that was being escalated and if the incident was identified to be a software problem, was the call then rooted to the System Support Centre, the SSC –
Charles Cipione: Yes.
Mr Beer: – the third level of support?
Charles Cipione: Yes.
Mr Beer: Was that third level support, the SSC, responsible for resolving incidents promoted by the System Management Centre?
Charles Cipione: Yes.
Mr Beer: That resolution was recorded or ought to have been recorded in the PinICL system; is that right?
Charles Cipione: That is right.
Mr Beer: The maintenance of the PinICLs was the responsibility of the System Support Centre through resolution and closure, with communications back – right back to the beginning – to PowerHelp; is that right?
Charles Cipione: That is correct.
Mr Beer: If additional evidence was required, that’s to resolve the issue, the third level of support, System Support Centre, would be engaged to gather the evidence – sorry, the second level would be engaged?
Charles Cipione: Yes, the second level. Yes.
Mr Beer: In the event that the SSC, third level, needed assistance from third-party vendors, would they escalate calls to the fourth line support, which dealt with technology from outside suppliers?
Charles Cipione: Yes.
Mr Beer: Can we turn, then, to PinICLs and PEAKs over the page, please. Each incident, you tell us, logged in the PinICL system is referred to as a “PinICL”; is that right?
Charles Cipione: Yes.
Mr Beer: The only ostensible difference between a PEAK and a PinICL is what?
Charles Cipione: The PEAK system – there was a PEAK system that replaced the PinICL system using updated technology. There was a big difference in the format of how they were recorded in the technology that underpinned them but, essentially, they carried out the same function.
Mr Beer: You say:
“As new PPs …”
That’s PinICLs or PEAKs; is that right?
Charles Cipione: Yes.
Mr Beer: “… are logged by a Team Member they are assigned a unique reference number.”
Is that right?
Charles Cipione: Yes.
Mr Beer: Which sets out when it was open, the last update to it, its open or closed status, a summary of the issue, and the product group. We’ll see one of these in a moment.
Charles Cipione: Yes.
Mr Beer: If available, information is captured relating to work packages, fixes, other PinICLs or PEAKs for reference; is that right?
Charles Cipione: Yes.
Mr Beer: The way that both PinICLs and PEAKs are set out is chronological and begins with the team member describing the issue, assigning a call priority, a call type, an estimated completion date, and routing to a team leader who would or should review the call, providing approval or rejection, and then rooting the call back to the relevant team member; is that right?
Charles Cipione: Yes.
Mr Beer: Can we deal with KELs, please? Separately ICL Pathway and Fujitsu maintained a knowledge base of information that included known issues in the IT system; is that right?
Charles Cipione: Yes.
Mr Beer: That knowledge base was known as the Known Error Log or KEL?
Charles Cipione: Yes.
Mr Beer: An individual entry is referred to as a KEL; is that right?
Charles Cipione: Yes.
Mr Beer: They contain information on how to address or rectify issues that have previously been identified within the system?
Charles Cipione: Yes.
Mr Beer: The responsibility for maintenance of KELs rested with third and fourth line support; is that right?
Charles Cipione: Yes.
Mr Beer: They are often, or sometimes, referred to during the resolution of a PinICL or a PEAK?
Charles Cipione: Yes.
Mr Beer: They contain structured attributes including the type, summary, the open or closure date, the status of the KEL and visibility. What does “visibility” mean?
Charles Cipione: How broadly it’s been seen by the team.
Mr Beer: By the team or by anyone other than the team?
Charles Cipione: Well, everyone had access to the KEL system within ICL Pathway.
Mr Beer: They contain, in the body, information covering the symptoms, problems, solutions and related evidence.
Charles Cipione: That is correct.
Mr Beer: Can we turn, lastly, then, to some of the materials provided to you and look at some examples. By way of primary materials – and these were date limited up until and including the rollout – I think you were provided with extracted IT incident tickets, PinICLs, yes –
Charles Cipione: Yes.
Mr Beer: – from Fujitsu’s PinICL system, yes?
Charles Cipione: Yes.
Mr Beer: Equivalent extracted tickets from the PEAK system; is that right?
Charles Cipione: Yes.
Mr Beer: Two archived PinICL databases, yes?
Charles Cipione: Yes.
Mr Beer: I think we’ll hear in a moment that, in fact, you didn’t analyse, in agreement with the Inquiry, to any extent the archived databases that you were provided?
Charles Cipione: Yes.
Mr Beer: Extracted records from the KEL system – is that right – so a series of KELs –
Charles Cipione: Yes.
Mr Beer: – and a collection of monthly reports; is that right?
Charles Cipione: That is correct.
Mr Beer: Can we start with the PinICLs and PEAKs, then. Can we look, please, at page 167 of your report. You explain – this is an appendix to your report – an example PinICL; yes?
Charles Cipione: Yes.
Mr Beer: You explain in the rubric at the top that this is an example PinICL with some of the challenges of interpreting this highlighted as call-outs. Do you mean, by “call-outs”, the boxes on the side, which have an arrow back to the underlying text?
Charles Cipione: That’s what I’m referring to.
Mr Beer: Thank you. You say you deliberately selected this as an example PinICL because it contains more fulsome descriptions and many PinICLs are more challenging to interpret?
Charles Cipione: Yes.
Mr Beer: You say, by way of note, that a PinICL was written for internal tracking purposes by the team trying to investigate and resolve the issues, not written in a way to give a complete and accurate explanation of all investigatory steps, such as that somebody reviewing them 20 years later could fully understand what had occurred?
Charles Cipione: That is correct.
Mr Beer: Can we blow up the PinICL in 18.6, please, so it takes up more of the page. Thank you.
So PinICLs recorded in a standard format certain information; is that right?
Charles Cipione: That is correct.
Mr Beer: It’s gone back. Thank you.
There’s a number at the top, PinICL Expor and then PC0044570. Was that a reference number, a unique reference number?
Charles Cipione: Yes.
Mr Beer: There then is a “Summary”, can you see the word “Summary”? Just underneath the reference.
Charles Cipione: Oh, yes, yes.
Mr Beer: Was that, or meant to be, a summary of what the issue was?
Charles Cipione: Yes.
Mr Beer: You can see there’s an opened date, this one at 4.48 on 3 May 2000, I think that is?
Charles Cipione: That is correct.
Mr Beer: Then the last update and status, you can see that it was last updated at 5.21 on 6 July, and that its status was closed?
Charles Cipione: Yes.
Mr Beer: The customer, do you know who that referred to? Not who that gentleman was, but what the – who the customer was in that field?
Charles Cipione: Um, generally it was the person generating the call to the helpdesk that – if this got promoted – if it came through the helpdesk, you know, the customer that reported it.
Mr Beer: Then on the right-hand side, the product group and the products at fault. Was this meant to identify the two things that are set out there, in this case the product group was the EPOSS, the electronic point of sale system, and the desktop and the product within that as the cash account?
Charles Cipione: Yes.
Mr Beer: Would there be a similar short descriptor on other PinICLs?
Charles Cipione: Theoretically, yes. I did not – I do not think that everything was filled in all the time but I don’t remember that exactly. But I do not believe that – but generally, it was filled in.
Mr Beer: Yes, I mean, you say that you’ve picked a fulsome one, which has got all of the information that we might want to see.
Charles Cipione: Yes.
Mr Beer: In the reference section, what was that for? Can you see that, the box on the left-hand side, a third of the way down the page?
Charles Cipione: So this had, you know, this had different extra information to the extent that they were available. So the copy from – you can see that even above, in the summary, it’s indicating that this PinICL was a copy of PinICL PC0044007 that’s repeated here, with the indicator that it was a copy from. The SSC KEL, that – that’s the KEL – a KEL reference that could be related to this that’s shown here. The other, I’d have to look more closely and remember what –
Mr Beer: What that code is?
Charles Cipione: Yes.
Mr Beer: Then to the right, a products table. What was the purpose of the products table?
Charles Cipione: So this is actually repeating the information that we went through also. The product group, “EPOSS and DeskTop” and the product name “Cash Account”, which was shown in the header.
Mr Beer: Then the activities log, which takes up the rest of this page and, in fact, I think if we looked at this one, we’d find it went on for pages and pages and pages. We can see this page 1 of 8 in the bottom right-hand corner.
Charles Cipione: Yes.
Mr Beer: What was the activities log?
Charles Cipione: This is the running dialogue of the process of analysing and hopefully remediating this particular issue in the PinICL.
Mr Beer: It proceeds, is this right, in chronological order with earliest entry first and latest entry last?
Charles Cipione: Yes.
Mr Beer: In some of the added boxes, the call-outs, you have noted three things: firstly that on this one, there’s an example of some typos, shorthand and acronyms that are used that you as a reviewer would have to decipher?
Charles Cipione: Yes.
Mr Beer: Can you just explain the significance or relevance of the second call-out explanation?
Charles Cipione: So the current release version of the system, so this indicates to me that the dialogue here believes that the new release of Horizon, the CSR-C13R, should – to me, this indicates that this problem should be resolved when this release of the software had been implemented.
Mr Beer: Then, lastly, you note an entry at 8.11 in the morning on 4 May, “Call transferred multiple times within a ticket”. What was the significance of that?
Charles Cipione: This just to highlight the fact that you’ll see the username changing throughout the chronology of this particular PinICL and we’re just showing you where that particular change was initiated. So, in this case, Richard Coleman was assigning the PinICL to Garrett Simpson.
Mr Beer: I think you were provided with 56,489 of these –
Charles Cipione: Yes.
Mr Beer: – as individual PDFs –
Charles Cipione: Yes.
Mr Beer: – in a date range of 7 July 1996 and 31 December 2000?
Charles Cipione: Yes.
Mr Beer: Thank you. Can we turn to PEAKs, please, and look at an example PEAK over the page, at page 168. Again, if we can just blow that up so it’s a bit clearer. Thank you.
So we know that in 2003, ICL replaced PinICLs with the PEAK system. Is it right that the PinICL system was archived and any open tickets from the PinICL system were transferred/migrated across to the PEAK system?
Charles Cipione: That is correct.
Mr Beer: The purpose of the PEAK system served a similar function to the PinICL system, and therefore, as you say in your report, the origins of the tickets within it would be much the same as those identified for the PinICL system –
Charles Cipione: Yes.
Mr Beer: – ie the sources by which these things would be raised?
Charles Cipione: Yes.
Sir Wyn Williams: Sorry, Mr Beer, I am a little lost. We’ve gone over the page, as I understand it but, unless I’m lost, this is still a PinICL. My example is of a PEAK is on page –
Mr Beer: 172. Quite right, sir.
172, please. Thank you.
You were telling us that it performed a similar, if not identical, function and, therefore, the origins, the sources of it, would be the same?
Charles Cipione: Yes.
Mr Beer: Can we see similarities in layouts between a PEAK and a PinICL?
Charles Cipione: We definitely can see similarities. Much of the tagging information that you saw in the PinICLs exist in the PEAKs and, importantly, the running dialogue is in the body, in the activity body of the PEAK, just as it was in the PinICL.
Mr Beer: But set out slightly differently?
Charles Cipione: I’m sorry?
Mr Beer: But set out slightly differently?
Charles Cipione: Yes, yes. The format is different.
Mr Beer: Can you just talk us through the four or five lines at the head of the PEAK, and what information we’re being provided with there?
Charles Cipione: Certainly. So we see the call reference, a unique identifier for this PEAK. The release for resolving this PEAK appears to be targeted at CSR-C13_2R. The call type that has been assigned to this PEAK is a live incident. The group that is working at this particular issue is the EDSC, with a target date resolving this of 19 May 2000.
Mr Beer: You make the point in the box at the top there that when you were interpreting these PEAKs, as with the PinICLs, there were acronyms that you had to look up to find out what they meant?
Charles Cipione: Yes.
Mr Beer: Yes, and then on the right-hand side?
Charles Cipione: So on the right-hand side, the first call-out box talks about – this ticket is referring to another call, which indicates that they’ve seen this issue before, and the second call-out box is the ticket is advised to close, then gets reassigned, but no detailed explanation. It’s a bit of confusion for me, as a reader, to, you know, understand exactly what’s going on based on this description.
Mr Beer: I think you were provided with 13,442 – I’m sorry, 16,530 PEAKs; is that right?
Charles Cipione: Yes.
Mr Beer: You found 13,442 duplicated PEAKs in the PinICL population; is that right?
Charles Cipione: Yes.
Mr Beer: So what was to done as a result of that?
Charles Cipione: We picked the PinICLs to use for our processing.
Mr Beer: You say that another issue concerned you, and that was the ordering of the activities table in the PinICLs; is that right?
Charles Cipione: Yes.
Mr Beer: What was the difficulty there?
Charles Cipione: As the original population of PinICLs were delivered to me and my team, it became apparent that the actual chronology shown within the activities log was not in actual chronological order, and it made it obviously very difficult to read.
Mr Beer: If we can go over the page in your report, that’s page 60 we were at, thank you. At the foot of the page, I mentioned letter that you didn’t examine the PinICL archive databases that were provided to you, in agreement with the Inquiry. At the foot of the page, you deal with the summary of the PinICL and PEAK data used to undertake your review, and you say that this dataset changed over the course of the review as you receive multiple copies of the same or very similar data across different deliveries. You had to make decisions as to what datasets to use.
You also had two analysis workstreams which were at different states of progression when some of the additional data was provided. You decided that these workstreams should, in some cases, use different datasets. You describe the two workstreams as “Analytics” and “Document Review”. Can you explain what you mean by those, please?
Charles Cipione: Certainly. So the analytics review was trying to – was using some of the data elements that we called out when looking at the example, PEAKs and PinICLs. Not the dialogue, necessarily, but some items that were captured from within the dialogue, or within the header information of those documents. Those were used simply for grouping and summing and creating charts based off of the information in those structured data components.
A document review workstream, on the other hand, existed with the intent of identifying different groups of individual PEAKs or PinICLs to be reviewed manually. So in other words, we wanted to read everything – we wanted human beings to read everything, to gain as good an understanding of the documents as possible.
Mr Beer: I think you summarised, in the table at 6.1, the documents used in your review by reference to the series of documents that you received and the use to which they were put; is that right?
Charles Cipione: Yes.
Mr Beer: So if you start in the middle column, the reference and description, the A1, A2, A3, A4, A5, B1, B2, are the series of documents that you received –
Charles Cipione: Yes.
Mr Beer: – in the course of your review. The total documents provided the use to which they were put?
Charles Cipione: Yes.
Mr Beer: You say that neither the analytics nor the document review workstreams were used in relation to the first dataset, A1, because it was agreed with Fujitsu that these data contained errors and were therefore replaced by documents at A4 – is that right –
Charles Cipione: Yes.
Mr Beer: – and A2, as it was recommended by Fujitsu that you use the PEAK versions of these, which were contained in the dataset B1; is that right?
Charles Cipione: Yes, that is right.
Mr Beer: If you turn over the page, please, to paragraph 6.2.25, can you explain 6.2.25, please? Can you explain the discovery that you refer to there, please?
Charles Cipione: So this is simply explaining that while the PEAK system did replace the PinICL system, the archive databases that were received contained all of the information that was in the PEAK system. So it was appropriate to use these archive databases for purposes of our analytics workstream.
Mr Beer: Thank you. Can we turn to Known Error Logs, please, and can we turn up page 174 of your report, please. Is this is an example of a KEL, a Known Error Log?
Charles Cipione: Yes.
Mr Beer: You explain that this was a knowledge management tool used by both ICL Pathway Limited and Fujitsu to explain how to deal with, or to work around, issues that arose in relation to the Horizon System. Can you talk us through it, please, using this as an example?
Charles Cipione: Yes, certainly. So to the extent that an issue – that Pathway was aware of an issue, they had this mechanism to record, you know, the – you know, some general information which is contained in the header, you know, that – the type, the title, the summary. You know, key release dates, keywords that are associated with it.
It also describes the symptoms that describe the particular issue, as well as what they believe the problem is. Importantly, they also provide some text for what the helpdesk should communicate back to any user reporting a similar problem. So in this case, the solution that the helpdesk should recommend to the user is to reboot the counter and, if the message reappears, then send to an engineer.
So what – the purpose of a KEL is, you know, to document known errors, but to also provide remedial, or recommendations to users reporting similar problems.
Mr Beer: In terms of the header, “HORIZON KEL rcoleman3549n”, in your report, you say this contains the metadata for the KEL, and it included the name of the Fujitsu employee who raised it, the identifier for the PEAK or PinICL that originated the KEL and the version number of the KEL; is that right?
Charles Cipione: That is correct.
Mr Beer: That’s all in that top line, “HORIZON KEL”?
Charles Cipione: Yes, you can see that the PEAK information is – the PEAK identifier is the bit below.
Mr Beer: The symptoms, is this a fairly typical description of symptoms that we see here?
Charles Cipione: Typical, in terms of –
Mr Beer: In terms of length and detail?
Charles Cipione: Yes.
Mr Beer: Is this a description from the perspective of the reporting person, rather than the person to whom it has been reported?
Charles Cipione: On reading this, this looks like a Horizon-centric description of the problem.
Mr Beer: Not something that the subpostmaster might –
Charles Cipione: No, no.
Mr Beer: So you’ve explained what the problem and the solution was, or what those parts of the KEL were intended to achieve. In terms of distribution or availability, viewability of the KEL, is it your understanding that the KEL system was available in read-only mode to the first line support, that’s the Horizon Helpdesk?
Charles Cipione: Yes.
Mr Beer: The second line support, the System Management Centre?
Charles Cipione: Yes.
Mr Beer: The third line support, the System Support Centre?
Charles Cipione: Yes.
Mr Beer: That third line support, as well as having read-only access, was permitted to create a new KEL themselves or to make changes to an existing KEL, to update a KEL?
Charles Cipione: That is correct.
Mr Beer: You say in your report that you noted that not all sections of a KEL were completed in all of the KELs, which indicated to you that not all of the sections were mandatory when either creating or updating a KEL; is that right?
Charles Cipione: Yes.
Mr Beer: Is it right that it’s your understanding that the KEL system, the Known Error Log system, was replaced in about July 2009 by Knowledge Base or KB, a new system?
Charles Cipione: Yes.
Mr Beer: You had, I think, 656 KELs produced to you in HTM format; is that right?
Charles Cipione: In HTM, yes.
Mr Beer: They were between the dates of 26 May 1998 and 28 December 2000 –
Charles Cipione: Yes.
Mr Beer: – and is the last species of principal documentary information that’s provided to you a management report?
Charles Cipione: Yes.
Mr Beer: You had 105 of those given to you, and I think they included 19 Pathway Programme monthly reports summarising the business development activities of the Pathway Programme –
Charles Cipione: Yes.
Mr Beer: – 13 monthly joint implementation reports?
Charles Cipione: Yes.
Mr Beer: They’re jointly issued by ICL Pathway and Post Office Counters Limited?
Charles Cipione: Yes.
Mr Beer: All Pathway customer service reports containing summaries of the performance of the ICL Pathway Customer Service Business Support Unit?
Charles Cipione: Yes.
Mr Beer: Then the largest category, 44 Pathway monthly reports, being a comprehensive management report for ICL Pathway ranging from October ‘96 to December 2000?
Charles Cipione: Yes.
Mr Beer: Plainly, you didn’t get every report for every month in that time range?
Charles Cipione: That is correct.
Mr Beer: The two managing directors were the approval authorities for those reports, JH Bennett, Mr Bennett, was the approval authority up until November 1999, and M Stares. Mr Stares was the approval authority beginning in January 2000?
Charles Cipione: Yes.
Mr Beer: They followed a typical format; is that fair to say?
Charles Cipione: Yes.
Mr Beer: You set out the ten parts of them in Roman numerals in paragraph 6.4.1(i) to (x) of your report?
Charles Cipione: Yes.
Mr Beer: You focused on that largest category, the 44 reports, the Pathway monthly reports?
Charles Cipione: Yes.
Mr Beer: Four of them were duplicative, and therefore you had 40 reports to analyse?
Charles Cipione: That is correct.
Mr Beer: Mr Cipione, they’re the only questions that I ask of you. I’ve had a question fed through to me. If you just give me one moment. (Pause)
In fact I’m not going to ask that question now. I think that’s for next time.
Charles Cipione: Okay.
Mr Beer: They’re the only questions I ask of you at the moment. You’re returning to us, I think, on 18 and 19 November. Ordinarily, the chairman would say to you that you’re in purdah, that you shouldn’t discuss your evidence with anyone, or the evidence that you are to give, with anyone.
Can I ask for permission, sir, to be released from that embargo and so that Counsel to the Inquiry can speak with Mr Cipione in the intervening period, including about his evidence?
Sir Wyn Williams: I’m very much inclined to accede to that request. I note that there are a number of CPs in the room. If anyone wishes to oppose that suggestion, now is the time, otherwise I’m going to allow Mr Beer to do as he suggests.
Mr Beer: Thank you very much.
Sir, we said to Core Participants that now, on this occasion, might not be the opportunity to ask Mr Cipione questions, that they may wish to hold those over until the November sessions. I think all Core Participants have done so, in that we haven’t received any questions that other people wish to be asked on this occasion.
So, sir, unless you had any questions of Mr Cipione, that’s the end of his evidence for today.
Sir Wyn Williams: No, I don’t have any questions, Mr Cipione, and I would just like to thank you for the clarity of your written and oral evidence to date, and look forward to seeing you again in a few weeks’ time.
The Witness: Thank you, sir.
Mr Beer: Sir, thank you very much. That means that we’re not sitting tomorrow. We won’t interfere with the order of witnesses because they’ve all been warned for specific periods of time over the coming months, and therefore we sit again at 10.00 am on Thursday.
Sir Wyn Williams: No, I agree with that. I think we should stick to the timetable unless we’re beginning to lose time. When we’re gaining time, I think we can use it for other useful purposes, Mr Beer.
Mr Beer: Thank you very much, sir. Can I thank you also for spotting the deliberate error earlier on when I –
Sir Wyn Williams: I think that was a test to ensure that I wasn’t too remote from the proceedings, Mr Beer.
Mr Beer: Very much so. You got me, sir. Thank you. So 10.00 am on Thursday.
Sir Wyn Williams: Yes. Well, actually, since it’s highly likely that I will appear in person on Thursday, could we just say 10.15 so as to avoid any possibility that people will have to wait for me?
Mr Beer: Sir, thank you very much. Yes.
(4.08 pm)
(The hearing adjourned until 10.15 am on Thursday, 20 October 2022)