23 February 2023 – Alan D’Alvarez
Hide video Show video
(10.00 am)
Mr Blake: Good morning, sir. Can you see and hear me?
Sir Wyn Williams: I can indeed, yes.
Mr Blake: Thank you very much. I’m going to call Mr D’Alvarez, please.
Alan D’Alvarez
ALAN D’ALVAREZ (sworn).
Questioned by Mr Blake
Mr Blake: Thank you very much. Can you give your full name please.
Alan D’Alvarez: Alan George D’Alvarez.
Mr Blake: Mr D’Alvarez, you have previously given evidence to this Inquiry. Your witness statement is WITN04800100 and that’s already gone into evidence. Do you have a copy of that in front of you? It should be behind tab A.
Alan D’Alvarez: Yes, I do.
Mr Blake: You’ve confirmed the truth of that statement before but just to confirm once again, does that remain true to the best of your knowledge and belief?
Alan D’Alvarez: It does.
Mr Blake: Thank you. As I say, because you’ve given evidence, I’m not going to go into detail about your background, we already know about that. Just one question about your background is really when you first became involved in what we know as Horizon Online or HNG-X?
Alan D’Alvarez: Horizon Online, I got involved in 2009.
Mr Blake: The reason I ask is I just want to bring one document to your attention, it’s FUJ00116732. The document itself isn’t really important, it’s a PowerPoint presentation but it’s page 7 of that document. It’s the fourth entry on page 7. Thank you. If we could zoom in, it has your name there. These are documents that are reviewed and it says there was a previous independent review by A D’Alvarez, July 2007; do you remember that involvement at all?
Alan D’Alvarez: Yes, I do.
Mr Blake: What was that?
Alan D’Alvarez: I was, at that time, on assignment in the USA, and on my – when I was back in the UK for some meetings that we had, because of my previous engagement with Post Office, I was asked could I provide an overview because there were some issues, could I just talk to some people and just do a review in my two weeks that I was in the UK, take two of the four weeks I was in the UK, just to discuss the situation.
Mr Blake: Broadly, can you tell us what you concluded in that report?
Alan D’Alvarez: I concluded – so what I concluded in that report broadly was the solution that had been signed up to was different to what I recall when I was – that was being discussed with Post Office when I was still on the account back in 2005. That the approaches being used with regard to development were not appropriate, and they should look to do more of a kind of a classic approach, as opposed to an agile approach to development, and just gave some explanations as to what I believe needed to happen to bring the project under greater control.
Mr Blake: Do you know who asked you to do that report?
Alan D’Alvarez: I believe it was Lester Young, I believe.
Mr Blake: Were there concerns about HNG-X at that stage within Horizon, withing Fujitsu?
Alan D’Alvarez: Can I correct myself, I think it was Peter Jeram.
Mr Blake: Were there concerns within Fujitsu about how the project was working?
Alan D’Alvarez: It was late. It was already signalling that it was missing its key milestones.
Mr Blake: Thank you. I’m going to now take you to some documents that I took Mr Burley to yesterday. I don’t think you’ve seen all of Mr Burley’s evidence from yesterday; is that right?
Alan D’Alvarez: I only saw part of it.
Mr Blake: It may be that I’m repeating matters I went through yesterday but that’s for your benefit rather than for everybody else’s. Can we look at FUJ00092754, please. These are the “Notes of the Horizon Next Generation Joint Progress/Release Board” of 28 January 2010. Mr Burley was the chair and you attended that meeting; is this something you remember in broad terms?
Alan D’Alvarez: These were regular meetings that we had, yes.
Mr Blake: It’s page 3 of that document that I asked Mr Burley about yesterday, and it’s the second entry on page 3. It says there:
“The delay in the commencement of Volume testing means that we will not be able to perform a significant amount of testing before commencing the Medium Volume Pilot. Hence we will need a significant amount of data to be collected from the Live Branches and Data Centre. The data will also require careful and thorough analysis.
“AD to confirm how this will be achieved.”
Is it your recollection that there was a reduction in the testing before commencing the medium volume pilot, howsoever small it may have been?
Alan D’Alvarez: So that’s specific to the volume testing. So we had a volume test schedule and with the volume test schedule it went in stages in different parts of the system but it also blocked out to where we would go to 25 per cent volumes, 50 per cent volumes. The intention of all the testing, what we call laboratory testing, would be complete before we go to pilot and pilot’s live testing. So the intention was to complete all the laboratory testing before. We hadn’t completed all the volume testing.
We had completed an amount – I can’t recall where we got to, either 25 per cent or 50 per cent loads. It was assessed that to go into a pilot and to go to medium volume pilot, which I believe was around 250 post offices, that we stressed the system sufficiently. However, what we proposed was we would monitor – we were already monitoring the performance at the branches. What we would do we’d collate those because when you do laboratory testing it’s in sterile conditions. You don’t have the real world rounds and networks and that, so it’s kind of model testing. So to give us greater assurance, because we hadn’t completed up to full load testing, we’re going to take the performance statistics and compare them to our testing statistics or our test results to see whether what happening live correlates to what we see in testing, to give us that greater confidence.
Mr Blake: Would it be fair for me to say that that kind of testing that was carried out, or the analysis of the data, is less than was originally proposed?
Alan D’Alvarez: More. When I say the analysis of the data was more, so we hadn’t completed the testing. We did complete the test within the next four weeks, four or five weeks. We hadn’t completed the testing at that time. So going into medium volume pilot, it was – we hadn’t completed the testing that we had on the schedule. Was it of material impact? We didn’t believe so. So we did complete all the testing. We completed testing up to 50 per cent – 25 or 50 per cent load to equivalent to 3,000 to 6,000 post offices we’ve tested at scale, in our laboratories and the medium volume pilot was, I think, 250 or thereabouts.
Mr Blake: The reference in this particular entry about not being able to complete a significant amount of testing before commencing the pilot suggests that there was less testing.
Alan D’Alvarez: In the volume.
Mr Blake: In the volume.
Alan D’Alvarez: Only in the volume, not in the functional testing of the business application.
Mr Blake: Are there risks involved in reducing the amount of testing in the volume; howsoever small, are there risks in reducing that amount of testing?
Alan D’Alvarez: Our assessment from Fujitsu was no because we’d done sufficient testing for the amount of stores that we were going to bounce off – or post offices – were deploying in medium. However, as I’ve explained, because it was laboratory testing, we wanted to put in some additional controls just to make sure that what we’ve tested, the results in testing, correlate to what we were seeing in live, so give us greater confidence.
Mr Blake: Why would it be proposed originally as an idea?
Alan D’Alvarez: Proposed?
Mr Blake: Why was it proposed originally that there was more testing and that amount of testing was reduced or are you saying it simply wasn’t reduced?
Alan D’Alvarez: It wasn’t reduced. It was completed. We completed all the testing. It was the timescales in which we completed the testing.
Mr Blake: So they were over a longer period?
Alan D’Alvarez: Yes. So we continued testing the performance whilst we was in pilot. The original plan was to complete all our laboratory testing before going into pilot. Only on the performance, we continued to test whilst we was in pilot.
Mr Blake: Is there an advantage in completing that testing before the pilot?
Alan D’Alvarez: It allows the team to focus more on the pilot because, obviously, we wanted to clear everything that we had so we can focus all our attention on supporting the pilot. But I don’t believe it had any material impact on risk or anything going into the pilot.
Mr Blake: Thank you. I’d like to take you into the next document, that’s FUJ00097159. Again, it’s a document I took Mr Burley to yesterday. It’s a meeting of the same day, this time of the release authorisation joint board. Again, is that something you recall, those meetings?
Alan D’Alvarez: Yes, they were the meetings that we used to track our position against the various acceptance case.
Mr Blake: If we go over the page, please, and it’s the shaded section that I’m going to start with, it says:
“DC confirmed that there are no outstanding High Severity Acceptance Incidents and that all other thresholds are within tolerance for Acceptance Gateway 3.”
The third paragraph says:
“However it was agreed that the high priority fixes in ‘Reset 4’ …”
Do you remember what Reset 4 was?
Alan D’Alvarez: I believe that’s a release that we had, so we had a number of releases that we called, and that was – basically will be delivered as part – so we had maintenance releases that went out not just for the programme deliverables but also just for general maintenance patching and things, and then the Reset 4 would be the programme release that we were to put into that maintenance release.
Mr Blake: Thank you. So the high priority fixes:
“… to be delivered as part of [the] Maintenance Release … could constitute a High Severity [Acceptance Incident] if not delivered in time for the High Volume Pilot …”
As you said, it’s just over 200 branches.
If we scroll down, it says there that Mr Burley:
“MB offered an option to remove items from Reset 04 which are not regarded as High priority – if they are at risk of missing the High Volume Pilot deadline, or affecting the delivery of items which are High priority.”
Is it your recollection that Mr Burley and the Post Office were trying to make it easier and quicker to get the pilot up and running and speed things up a bit?
Alan D’Alvarez: It’s an option that’s discussed. So with all programmes you have to balance risk against progress. So one of the options that was discussed and was – we, I think we agreed to actually take away and understand what that actually meant with regard to risk and that, but one of the options were – there were some fixes that needed to go in and we couldn’t progress. Others were of less impact, some could be cosmetic. Some may be – and the bigger you make a release, the more risk you put on the timescales that that release will be complete, fully tested and ready to go.
So it was to – I think the discussion at the meeting was very specific to if we wanted to achieve the milestones, was there anything of less significance that could wait to a later release and would that – if we did remove that from that particular earlier release – would that derisk making a timescale?
So it was, you know, we’d – delivering a programme, you know, you’re always looking at the balance of risk and timescales.
Mr Blake: Absolutely. So you describe risk against progress. Was there, coming from the Post Office, a push – we’ve spoken about the delay in the programme, for example. Were they keen on making progress and accepting a greater degree of risk?
Alan D’Alvarez: It depends what part of the Post Office, really. So within the programme team, we were pretty much joined up at the hip, myself and Mark and our team and his team, in as much as we had a focus on – there are certain items of delivery that you had to get right before we progress. Others were up for risk assessment. I had pressure from my company. I don’t know what pressure he got but I’m sure he would have got pressure outside because people want to see progress.
Mr Blake: Was he making clear to you that he was under pressure to make progress?
Alan D’Alvarez: We had an approach where, when we had to potentially signal delays, we would work together to what is the messaging to our joint – you know, do we agree firstly amongst ourselves that a delay is the right thing? Let’s have look at the whole position, look at the risk. If we do agree amongst ourselves that a delay is the appropriate course, then we would work on the messaging to our respective organisations because with that comes quite a lot of disruption in the organisation, quite a lot of disappointment, and what – we wanted to make sure that the messaging was clear.
So in discussions, it’s the normal pressures of a large programme. People want to see it succeed. People want to see it go out, and it’s our job to navigate to make sure that we’re protecting – you know, that the risk is being managed appropriately.
Mr Blake: If we stick with this document and move to the penultimate page, page 9, there are a couple of issues that are mentioned at the bottom of page 9 and into page 10. These are new actions, 28 January 2010. The first is:
“Branch Trading Statements at Coton House and Warwick. It was confirmed that on screen error messages had occurred at both offices which meant that their Branch Trading Statements could not be completed.
“The root cause position at Coton House was understood, but the Warwick office had additional complexities.”
Just pausing there, do you remember this particular issue?
Alan D’Alvarez: I remember there were three issues. We refer to them as the CWD, I think it was Coton, Warwick and Derby, and when we went into pilot and it’s the – 10 or 12 offices, I forget how many, but there were a small number of offices, we now moved into a live situation. So what we put in was a huge amount of support, huge amount of monitoring, because it’s still test, the pilot is still test, and there were three different incidents that come up and they are two of the three.
Mr Blake: Thank you. If we scroll down, “Double settlement at Derby”.
Alan D’Alvarez: Yes, that’s the other one. That’s the third.
Mr Blake: Can you tell us about that particular issue?
Alan D’Alvarez: That was of greater concern because that one had potential integrity implications. So I remember at the time – so I’m much closer to that one, so I took ownership of that because of the potential severity of that. The other two were, on the initial assessment, more to do with operational rather than integrity.
Mr Blake: Then looking below, it says:
“Decision – Postponement of the next 10 branches. Based on the lack of a known root cause for AG3.70 [the first of those two] and AG3.71 [the Derby issue] it was agreed that the next 10 branches should be postponed until the impact and way forward is fully understood.”
Alan D’Alvarez: Correct.
Mr Blake: We’ve seen that only a few days passed and that postponement was reversed. Do you remember that?
Alan D’Alvarez: I recollect that we quickly got to an understanding of the root cause of the Derby, and I recollect that we had a fix and that fix had to go in before we could progress because it was significant, because it was a data integrity issue. Geoff Butts led on the other two because he is my deployment migration manager.
If I remember – I might get these in the wrong order. I believe Coton was to do with the migration itself, ie the migration tooling not completing its data downloads and we had a workaround, ie we would do the download through the migration tooling from Horizon to HNG-X, and I believe we would do a – we – someone had written a script to check is it complete as a compare. If it’s not, we will repeat the downloads to complete it.
So we still went through the migration tooling, so where we were satisfied that whilst we had to fix the migration tooling, that we could avoid that happening again.
The Warwick one, I believe that was to do with reporting, in as much as the underlying data was correct but there was a report that was created and it wasn’t the report – figures in the report were incorrect.
Mr Blake: We’ll come to the report issue. I think ultimately actually the Warwick one involved a node as well and it’s a bit more complicated.
Alan D’Alvarez: Okay.
Mr Blake: We don’t need to go into the detail of that right now.
Alan D’Alvarez: However, but I think by the time – within – you know, we worked over the weekend on this. I do remember that everyone was in over the weekend and we satisfied ourselves that we had fix for the Derby one, which was the critical one, and we had, I believe, appropriate way forwards or we had proposed, because we can only propose potential workarounds, it’s for Post Office to accept whether they’re acceptable or not. So we had a proposal, which I believe was accepted.
I don’t know how quickly we went into, but it wasn’t long, two weeks later we went.
Mr Blake: So the postponement was largely or significantly because you recognised the importance of matters that impacted on data integrity?
Alan D’Alvarez: All three we need to assure ourselves that, you know, but certainly data – that there was a way forward that would – that we could mitigate any impact. But certainly data integrity was the postponement, yes.
Mr Blake: Can we look at FUJ00092875, please. It’s page 3 of that document. It’s an email from yourself, which addresses the decision to deploy HNG-X to a further ten branches. This is 3 February now so it’s very shortly after you explained there was work over the weekend. It says there:
“The decision has been taken to deploy HNG-X to a further 10 branches with the migration button being pressed tomorrow for migration to complete Friday.”
There were, however, still two outstanding issues, the first is the branch trading statement issue, and it says there:
“This is where the in day migration process that happens once a branch hits the migration button is not correctly migrating across the summary data. This data is used to produce the branch trading statement”, et cetera.
The second issue being counter pauses in live:
“Yesterday and today a number of branches reported ‘screen freezes’ whilst operating HNG-X.”
Alan D’Alvarez: Mm.
Mr Blake: So is it fair to say that you were very much aware that there were still outstanding issues that you needed to get to the bottom of but the decision was taken to deploy HNG-X to a further ten branches because you were sufficiently satisfied with the work you had done over that weekend?
Alan D’Alvarez: We proposed that – we make – well, we jointly looked at the position with the joint programme team and our proposal was to – we satisfied ourselves the risk was manageable.
Mr Blake: Thank you. Your evidence is very much about joint decisions and joint working. Was there a difference in opinion at all on these kinds of issues between Fujitsu and the Post Office?
Alan D’Alvarez: Sometimes. On occasion there would be – we would propose something but Post Office would have the final say.
Mr Blake: In what sense?
Alan D’Alvarez: In a sense that if we felt that the risk was manageable to move forward, and that was our proposal, we had a workaround to an issue or something, Post Office, if they say they’re not comfortable, they’re the customer. It’s their business. They have the final say on programmatic decisions.
Mr Blake: It’s paragraph 4 of this email that I took Mr Burley to yesterday and I will read that for you. It says:
“We had a meeting with Post Office this evening which Mark Burley led from the Post Office side. Post Office are desperate for a date to start planning/rescheduling medium volume pilot. They accepted our position that we were not able to give this today. I expect that Mark will be keeping Dave Smith briefed and my reading is that if we are not in a position to give a target date by [close of play] tomorrow it’s likely to result in an escalation to Mike Young.”
Certainly the impression that’s given there is that pressure is being put on Fujitsu to get on with things and that, if they don’t, then it will be escalated. Am I wrong to form that impression?
Alan D’Alvarez: Get on with things. So the situation is specific here, is – so Post Office were responsible for the business change activities associated with the programme and we’re now into live pilot and, therefore, there’s a lot of communications, planning, you know, lining up of post offices. So “desperate” is my word, you know, that’s my word, in as much as, you know, being a qualified programme manager, and understanding the pressures that they’ll be under, is that what we had is a situation where we halted the pilot, we’re going cautiously into the continuance of the low volume pilot, which was always intended to be around 20 offices.
They would have had all the communications, all the countdowns, people lined up for the medium volume pilot because you do that many days before, like 45 days before there’s a big chain. Now we’re into a set of uncertainty where we’re signalling in to both organisations and they’re having to signal in to their business that they’ve paused and they cannot confirm dates of when we can line up so the Post Office would have been communicated to some of those that they had dates. They now had to be communicated to, to say they’re paused, but we can’t tell them when because we haven’t yet got to the root causes of those problems. So it wasn’t a case of we just need to get on to do the pilot; we needed to understand how long would it take us to resolve those problems.
So once we get to the root cause we can then assess is it a quick fix? Is it a long, complex fix that’s going to take several weeks? And we need to give – you know, Fujitsu needs to give Mark and his team an indication as to what are their planning assumptions they can make so that when they communicate to post offices, as opposed to leaving them, we’re going to delay the deployment but we don’t know when to, they can be more certain as to kind of provide that.
It kind of – it’s better when you’re communicating out that you can give someone alternative dates or a clear understanding of expectations. At this point we couldn’t give Post Office an expectation of how long it will be before we can go into medium pilot and that was the key. So for me, my take, he needs to know that because if he doesn’t, the business will start to ask questions, it goes up for their management, it goes to their management, and it creates a lot of distraction.
Mr Blake: Mr D’Alvarez, did you hear Mr Burley’s evidence on this or have you been told about Mr Burley’s evidence –
Alan D’Alvarez: I only saw about half hour of his evidence, and it was really around Horizon, which surprised me because I didn’t recognise him working on Horizon.
Mr Blake: You were quite careful today to distinguish the word “desperate” as being your word rather than his. Is that something you’ve given some thought to?
Alan D’Alvarez: It’s just me, you know, just saying that he’s desperate because you asked me because there pressure.
Mr Blake: Yes.
Alan D’Alvarez: That’s my, you know –
Mr Blake: The impression that you’re giving today is that it was quite a calm atmosphere, and –
Alan D’Alvarez: (The witness laughed)
Mr Blake: – there wasn’t pressure being brought, despite –
Alan D’Alvarez: No, there was a lot of pressure. A lot of pressure.
Mr Blake: Where was that pressure coming from?
Alan D’Alvarez: Well, it comes from within because we have committed to a plan, so – and then we have to explain when we, we’re not making key milestones that impact both organisations, and also, now because we’re in pilot, we’re not – you know, it’s not just impacting the people working on the programme. Delays and issues impact the business. So that there’s the pressure to get it right, there’s a pressure to be clear in communications and there’s a pressure to, you know, successfully deliver the programme.
Mr Blake: So although “desperate” itself is your language –
Alan D’Alvarez: Yes –
Mr Blake: – you would accept that there was pressure coming from the Post Office to get on with the pilot?
Alan D’Alvarez: And from Fujitsu, yes.
Mr Blake: In terms of an escalation to Mike Young, what did that involve?
Alan D’Alvarez: It’s my assessment that I needed to brief my senior executives because it could very well, that if, by telling post offices that the schedule times were being delayed, not giving them further information as to alternatives and that, things typically go up the chain, up their management. They would go up, it would go to my – it would go to someone who is sitting over the programme and then they would call my executive.
So it’s a briefing to say “Be prepared, this is the position, you may well get a call from Mike Young because he may well get an escalation, either from within his programme or from outside his programme”.
So for them to be aware of the situation and not be seen not to understand what’s happened. And also I needed to brief Fujitsu as well, because it’s a delay which causes us pain.
Mr Blake: Pain in what sense?
Alan D’Alvarez: Because we got teams lined up, time is cost.
Mr Blake: Can we look at FUJ00093056, please. This a “Horizon Next Generation Progress Joint Board Meeting” of 11 February 2010. Again, Mr Burley is the chair and you’re listed there as attending.
Can we look at page 4. About halfway down the page on page 4, we get to the BTS issues at Warwick again. They’re mentioned again and it says they’re new BTS issues in Warwick:
“BE to scan for the latest BTS reports from Warwick”, et cetera, et cetera.
So that’s just an update there in relation to the Warwick issue that we spoke, the branch trading statement issue.
Over the page, the final substantive entry in that table, there’s a separate issue. It says:
“Error message is seen at branches but not flagged up by FS [that’s Fujitsu] monitoring systems. The following message [appears]”, et cetera.
Was it typical, common, or do you recall instances where error messages may be seen at branches, but aren’t flagged up to Fujitsu’s own monitoring systems?
Alan D’Alvarez: So this was a new system. We’re going into pilot. The – not only – okay, so if we can just step back a bit. So the whole premise of HNG-X had three different elements to it, and I think the whole premise was the business processes didn’t change, so it wasn’t a business transformation; it was a technical transformation.
And it was based on – I think there were two key drivers. But there was a data centre element where we were migrating into Fujitsu secure data centres. There was a technical refresh of the back end and there was a refresh of the application.
So with all those we had to – you know, we didn’t – it’s not a case we’re putting a new application in and we’re tuning the system, or we have a system that’s already working; we have a whole new infrastructure and everything else. So we had to put in the monitoring and that and with monitoring, if you put too much – if you monitor everything, you can’t see the wood for the trees because, you know, you don’t know what’s important.
So it’s getting the tuning and the monitoring right. So in the early pilots, part of the pilot is if we’re – if there are issues that are being seen in the Post Office that impact the branches and we didn’t pick it up in our monitoring, we had to make sure our monitoring was extended for that. But what that meant was we didn’t get advanced sight before the postmaster got impacted or the clerk got impacted. So that would result in to helpdesk.
It may not – if we monitored that, we may not have been able to intercept it before the postmaster, I don’t know that particular issue but that was specific to making sure that our monitoring was properly tuned through the pilot.
Mr Blake: Irrespective of that particular issue, do you remember instances where error messages or other things might be seen by the subpostmaster that wouldn’t be flagged by Fujitsu’s own monitoring systems?
Alan D’Alvarez: That document reminded me of an instance. I can’t recall one – others, but there may well be.
Sir Wyn Williams: Mr Blake, I’m sorry to interrupt you but my screen is just showing “Post Office Horizon IT Inquiry”, and I’m not seeing either of you at the moment.
Now, I am. That’s fine.
Mr Blake: Let’s move on to FUJ00094192, we’re now at 18 February. It’s another meeting of the board.
Can we look at page 3. Towards the bottom of page 3, it’s again the Warwick issue. We can see there it’s shaded, and I mentioned earlier the issue of, I think – arranged for a base unit swapout at Warwick, so it seems as though the Warwick solution wasn’t just a software solution, it was a hardware solution; is that a fair interpretation? Node 6? Or is that – have I misunderstood that?
Alan D’Alvarez: I don’t know the reasons for that. That could, as I say, Geoff Butts was the lead on the Warwick one. Whether that was felt as an added precaution or whether that was felt as necessary, I couldn’t say.
Mr Blake: Are you able to assist us at all with the words there around not setting a precedent?
Alan D’Alvarez: We’re not setting a precedent that if you have issues, I’d expect that we don’t want to – every time there’s an issue in the Post Office, that we would go and swap out a counter. Because that would be not appropriate.
Mr Blake: It would be expensive, certainly?
Alan D’Alvarez: Not – and again, that leads me – those words lead me to conclude, but without any facts behind it, that that was just a secondary precaution, as opposed to a necessary step.
Mr Blake: Can we go over the page, please, to page 4, “Volume Testing”, so I think you’ve said volume testing was occurring during the pilot.
Alan D’Alvarez: Mm-hm.
Mr Blake: “LF reported that problem encountered to date with the first phase of volume testing have put the target end date of 20th February at risk.
“GA to reassure that correct resources are in place to resolve any problems as soon as possible.”
Then you have there LF – I think that’s Lee Farman of the Post Office – was confident that the correct focus is now in place. Is this something you remember at all?
Alan D’Alvarez: It’s just that we took the decision to do volume testing and volume testing is quite complex. So it’s very much stop/start.
Mr Blake: It seems as though it was closed because Mr – do you remember Lee Farman at all?
Alan D’Alvarez: Yeah, he was in the joint testing. He was one of their leads. I believe there was two, I can’t remember the other person’s name. If you told me I would. I believe he was on the non-functional side of testing.
Mr Blake: Was reliance placed on him and the Post Office to assure you that you could get on with things at that stage? So it seems as though it was closed because he was confident that the correct focus was in place. You’ve described a lot of your working relationship to be a joint one.
Alan D’Alvarez: Mm-hm.
Mr Blake: But certainly it seems to have been closed because he had confidence.
Alan D’Alvarez: Yeah. So he would work closely with my test lead, Debbie Richardson. They shared the same environment. They were both based in Bracknell, on the same floor, in the same area.
Mr Blake: But she’s not mentioned there. Is it because ultimately the decision to progress matters lay with the Post Office, or –
Alan D’Alvarez: Correct.
Mr Blake: Do you recall what the problems were that you encountered at that particular stage? It may be too specific a question because it’s 18 February 2010.
Alan D’Alvarez: No, I don’t recall. I don’t recall.
Mr Blake: If we scroll down that page, there is the reference to the “Trial Report/Final Balance Issue”. So this is an issue we addressed yesterday where the balance sheet prints the correct report in the trial report but not the final report, I think, and it says:
“PN to check if the proposed workaround is acceptable to the business.
“Permanent fix targeted for R01.08. However; this is dependent on the acceptability of the workaround, it may need to be a Hot Fix.”
Is this something you recall at all?
Alan D’Alvarez: Isn’t that the Warwick issue? It’s – it looks similar to the Warwick issue to me.
Mr Blake: I think it’s fair to bring to your attention the entry that is below that, which says:
“NOTE: for the avoidance of doubt, any workarounds have to be agreed by POL before implementation.”
Is that something you recall?
Alan D’Alvarez: Yes.
Mr Blake: Was that always enforced?
Alan D’Alvarez: Yes.
Mr Blake: If we go over the page to page 5, and the bottom three entries in that table, we have:
“MB [I think Mr Burley] thanked GA, DR and their teams for all their efforts in ensuring that the manual BTS fixes were successful applied as required.”
Do you remember at all what the manual BTS fixes were?
Alan D’Alvarez: No.
Mr Blake: Then the “Warwick Issue due to BAL node failure” is mentioned below that. Does that assist you at all with the Warwick issue?
Alan D’Alvarez: I can’t recall it.
Mr Blake: Can we move on then to FUJ00094268. This was an email chain I took Mr Burley to yesterday and it relates to the balance trading statement issue. It’s slightly difficult to read. I think if we start at page 10. Was this something that you saw in yesterday’s evidence or –
Alan D’Alvarez: No.
Mr Blake: – do we need to spend a bit of time on the actual content?
Alan D’Alvarez: No, I didn’t.
Mr Blake: Okay, so let’s look at page 10.
Alan D’Alvarez: This is in my pack though, I recognise this from my pack.
Mr Blake: There’s an email from Geoff Butts to Will Russell, Mark Burley, Barry Evans, et cetera. If we go over the page, we can see the issue there. You were copied into this email?
Alan D’Alvarez: Yes.
Mr Blake: Can I just ask you to read to yourself that paragraph and see if that refreshes your memory about what this particular issue is?
Alan D’Alvarez: So:
“This is to confirm the position on a fix for the BTS issue whereby printing the Trial Balance Report for BTS results in incorrect data being displayed in the Final Balance Report. A fix can be delivered and tested for inclusion within the 01.08 Maintenance Release as a counter fix. In the meantime, branches need to use the BTS Trial Balance Report, which is correct and discard the BTS Final Balance Report, which is incorrect.”
Mr Blake: Yes. Does that jog your memory about this particular issue?
Alan D’Alvarez: It does. This is what – Geoff was leading on the Warwick issue and I did keep abreast as to the progress on that.
Mr Blake: So you say keep abreast, some of these are sent to you, you’re copied into some.
Alan D’Alvarez: Mm-hm.
Mr Blake: What kind of a role did you play in this particular issue?
Alan D’Alvarez: Well, Geoff reported in to me, and so when we had – as I say, earlier in the pilot there are three significant issues that were raised: one was at Coton, one was at Warwick, one was at Derby, ie I said to Geoff I will take ownership of the Derby one that had – in the initial review of those incidents it had data integrity implications, and he took leadership because that was his role.
He was the person that was heading up deployments and heading up what we call Hypercare, ie the additional support given to pilot sites. He took the leadership or the lead role in resolving these issues.
Mr Blake: Can we look at page 9, please, which is a response to Mr Butts from Mark Burley. If we scroll down to page 9, he says there:
“Geoff
“Thanks but can I ask who you have agreed this workaround with? This is a legal document and there is a difference between a trial balance and a Final Balance. I would always expect CS to have a KEL for this – if agreed – to be able to explain the position to any subpostmaster who calls in.”
Do you remember this?
Alan D’Alvarez: I remember Geoff highlighting to me that the issue was wider and I gave Geoff the advice to make sure that our legal counsel was now engaged in making sure. So I believe this was Geoff looking at the incident as a technical incident, proposing a workaround, and then the response is “That workaround, are you sure because it’s a legal document?”
So I do remember Geoff raising it, I do remember specifically me saying to Geoff saying “Would you make sure that Jean-Pierre is now engaged on this?”
Mr Blake: Do you remember whether Geoff was concerned by the fact that it was a legal document?
Alan D’Alvarez: I wouldn’t say “concerned”, we knew that we now had to make sure that anything we proposed and go forward with, our legal people are happy with our proposal. But, again, ultimately I’d expect also Post Office to make sure their legal people were happy with any, if we were to go forward without a fix. So I don’t know what – I can’t recall how we actually went forward with this one but if we went forward with a workaround and not a fix, I would expect everyone to be consulting their legal counsel as this is a legal document.
Mr Blake: Would it be typical where fixes or workarounds affected what are described as legal documents, or documents used in legal proceedings, to be escalated within the company, as you say, to general counsel and others.
Alan D’Alvarez: We had a legal department so if there’s anything which we felt provided an issue with regards to either integrity or any other legal aspect to our system, we would engage – we should engage and my expectation is we did. I always engaged the legal and I had advised Geoff in this instance to engage with our legal team.
Mr Blake: Was there anybody else within management outside of legal who you would typically consult with in relation to those kinds of issues?
Alan D’Alvarez: I would certainly inform – at this stage, I believe, Gavin Bounds was on the account. He was the account business unit director. So I’d make him aware.
Mr Blake: Can we look at page 4 and scrolling into page 5, please. This is the – an email to Phil Norton from – I believe it may be from Gareth Jenkins, and it says:
“Phil,
“Alan D’Alvarez has asked me to respond to your concerns below.
“I’ll try and explain the issue and what has caused it.
“I assume you’ve seen the attached write-up of the issue which was sent to Barry Evans (and others) …”
Then there’s an explanation of the problem.
Do you remember asking anybody to assist with this particular issue?
Alan D’Alvarez: So I believe Geoff was on leave. Geoff wasn’t around because Geoff was leading on this. So Geoff was on leave. So I asked who was supporting Geoff in the investigations into that and was told that Gareth was. So I spoke with Gareth and asked him, “I got a response from Post Office, Geoff is on leave, could he please look at this and provide a response to Post Office?”
Mr Blake: What did you understand Gareth Jenkins’ role at this stage to be?
Alan D’Alvarez: So Gareth Jenkins’ role, he was always on the Horizon side and when we was migrating over to HNG-X, because I believe when we went live with Horizon, he – we’d become part of a customer services and, I believe, an architects group. There was an architects group, general outside of the programme. And his role was basically as a senior architect, he was acknowledged on the account as an expert on the counter applications.
So I’d expect that Geoff, because this is about balance sheets and things, to go to our expert on the account as to “Could you now look at this as an issue” because, as I said, what – in Horizon, what we did not do was change the business processes, or it was not a business transformation it was a refresh of the technology and how we actually supported their stuff.
So Gareth would be very knowledgeable as to what the consequence of balance trading statements are in respect of the business.
Mr Blake: He identifies there in paragraph 4 that there’s a bug in the way that the report is produced such that some of the in-memory copy of the data is overwritten when the trial BTS is produced.
Alan D’Alvarez: Yes.
Mr Blake: Do you recall discussing with Mr Jenkins the concerns within this email chain about the trial report being a legal document?
Alan D’Alvarez: I just recall discussing with Geoff that that position, because Geoff discussed it with me.
Mr Blake: Do you recall Mr Jenkins as being someone who was aware of the significance of a report such as that for legal proceedings?
Alan D’Alvarez: I’m not aware, but I – he’s our expert so I would have an expectation that he would be aware.
Mr Blake: Were you familiar at this time with his involvement in any criminal or civil proceedings?
Alan D’Alvarez: I understood from my original tenure on Horizon that he and others had given evidence.
Mr Blake: If we look at page 1, there is the email from Phil Norton at the Post Office to yourself. He has met with the Finance team to discuss the issue and he sets out deliverables that he would like, including:
“A complete and comprehensive list of all products where the volumes on the Final Balance … differ …
“A definitive statement detailing:
“How this defect has arisen”, et cetera, et cetera, including:
“A commitment to support POL in proving the integrity of the system in any subsequent legal action (specifically where the difference in the two reports is used as a means to challenge the integrity of the system).”
Is this something you remember being requested by the Post Office?
Alan D’Alvarez: I remember – well, I obviously received that e-mail. I don’t remember specifically at the – you know, from memory, but from my pack I remember I can see I received the email and I forwarded it to Geoff because, as I said, Geoff was leading on this particular one. And then also, in a previous conversation with Geoff I advised him to make sure that our legal team was fully engaged with any responses we gave on this subject.
Mr Blake: Why would Phil have sent it to you rather than to Geoff?
Alan D’Alvarez: Perhaps – well, I was the programme lead. I was the head person. So I assume Phil felt it significant enough to send it to the person that was leading the – you’d have to ask Phil.
Mr Blake: Was it typical to you to be asked by the Post Office to provide a commitment to support POL in proving the integrity of the system?
Alan D’Alvarez: No.
Mr Blake: Was this something novel?
Alan D’Alvarez: I wouldn’t say novel. It wasn’t something that was typical. When I saw that – as soon as I saw that, I recognised that we needed to engage our – you know, make sure that, as it is a legal document, anything that we propose, we ourselves assure ourselves that we’re not compromising Post Office.
Mr Blake: Did it concern you at all?
Alan D’Alvarez: On concern me, it concerned that with all issues that impact the end customer is a concern. This was a serious issue and that’s why we had, you know, appointed one of my senior people to take ownership to getting it resolved. There will be a number of bugs that will be raised during a pilot. Many of those just will be managed by the resolver groups, ie it goes through the process.
The serious of the Coton, Warwick and Derby was recognised and we put senior people on to make sure that we understood fully what was going on and our – and we had a proper resolution and that we were satisfied that the resolution was that the right resolution.
Mr Blake: Having seen in the previous months that there were these issues cropping up – bugs, as you described them – do you recall your reaction to being asked to support the Post Office in proving the integrity of the system in any subsequent legal action?
Alan D’Alvarez: My – I can’t recall my reaction but it would be make sure that our legal counsel understands that we had this request and anything we provide back, they need to assure.
Mr Blake: Would you have thought that you could prove the integrity of the system in any subsequent legal action?
Alan D’Alvarez: The – that’s an interesting question. Well, that’s a question which, because of the Derby incident, there was an incident in Derby in which there was a double entry, and that clearly had integrity issues. Because of that incident, I immediately went to my senior in the organisation and said, “We’ve received it” – and that was Maz Kostuch, so I worked in the programme project management organisation –
Mr Blake: Sorry that was who, sorry?
Alan D’Alvarez: Someone called Maz Kostuch. Maz Kostuch was the head of programme and project management for the public – private services division which Post Office was part of.
Sir Wyn Williams: (Unclear) – so that I make sure – I know there’s a transcript but I’d like to get that name accurate, if I could, please.
Alan D’Alvarez: Kostuch, K-O-U – I believe K-O-U-T-U-C-H, (sic) I believe that’s the spelling.
Sir Wyn Williams: Thank you very much.
Alan D’Alvarez: He was the person that was the head of programme project management and I said, we have an incident here, and it’s quite significant. He then engaged the lead technical person in the – who was – the name will come to me, I’m sure – and we agreed that we will get an independent review of the decision because there was two aspects which I was concerned about: one, there was that defect which was an integrity defect; and secondly, we didn’t pick it up in our testing, in our laboratory testing, we picked it up in the pilot, which is a test phase. And we needed to understand, “Well, hold on, if that happens, we can’t just say we fix that bug and just move on. Is there something inherent?”
So we arranged for some experts, application experts, there was two of them, I forget their names, immediately assigned them to this, dropped all their other work, and they undertook a review of the solution of the testing and gave a report.
Mr Blake: We’ll come to that report shortly –
Alan D’Alvarez: So it’s that report that led me to have confidence that the integrity of the system was good.
Mr Blake: We’ll come to some documents but do you recall the end result of this request from Phil Norton?
Alan D’Alvarez: It’s in my pack. I kind of briefly skimmed over it but it was a response that Geoff prepared, went to counsel, and I believe that was sent to Phil.
Mr Blake: Thank you. We’ll go through that now. Can we look at FUJ00094472. This isn’t the response itself, but this is – shows the sequence. These are further board minutes of the 11 March 2010. If we could turn to page 3 please. Thank you.
At the bottom there it refers to the “Trial Report/Final Balance issue”:
“PN to check if the proposed workaround is acceptable to the business”, et cetera.
But it’s the right-hand column that sets out the sequence, so we have 25 February:
“POL have requested that this be a Hot Fix as it is required before we migrate any further branches.
“Fujitsu to ensure deliverables listed in PN email … for current live branches are included.
“[4 March] GB has received feedback on integrity statement from Fujitsu legal. Info will be forwarded to PN.”
Then 11 March:
“PN has passed statement to P&BA who are reviewing with POL Legal team. PN will feed back to GB.”
This kind of collaborative approach on an integrity statement, is this something that you recall, is this something that was typical? Am I right or wrong to say that it was a collaborative approach?
Alan D’Alvarez: Yes, it was a collaborative approach. We worked as a joint team.
Mr Blake: Do you recall other collaborative approaches of this nature with regards to the drafting of an integrity statement?
Alan D’Alvarez: No.
Mr Blake: In terms of the resolution of this particular issue, irrespective of the ultimate issue that this branch trading statement may not ultimately matter for the sake of this Inquiry, but in cases of a bug of this nature, would you expect the Post Office to have told all branches, cascaded the information down or do you think that, a bug of this nature, it would be sufficient for it to be on a Known Error Log for subpostmasters to call in and, if they have a problem, that would be known on the Known Error Log?
Alan D’Alvarez: So, there’s two aspects to that. Firstly, we had to fix it. So if a Post Office hadn’t received a release, there was no reason to inform them. I don’t know how Post Office themselves communicated this. Would it be reasonable to advise those Post Office that were participating in the pilot test phase? That’s an advisable approach – I don’t know, we didn’t have that discussion – that they were responsible for communications to their business.
Mr Blake: What would your recommendation be, though, for something that affects the trading statement? How – looking at a recommendation for the future or something along those lines, if you come across an issue that has been phrased as affecting a legal document, the trading statement itself, do you think it is sufficient for it to be placed on the Known Error Log or do you think more action should be taken to draw that to subpostmasters’ attention?
Alan D’Alvarez: My understanding was that the workaround was to use the trial balance report.
Mr Blake: Yes.
Alan D’Alvarez: If that’s a workaround, that workaround will get communicated to postmasters because it’s a workaround, ie a workaround is there’s a known issue, to work round this issue, this is what we’re asking you to do. We’d also have it on the known error list because a postmaster may have forgotten about it or not read the advisory notice and, therefore, phoned up the service desk and it will be on a known error so they could then give that same advice.
But a workaround is very much something which, if it means asking the postmaster or their staff to do something different, that must be communicated to them for that workaround to be effective.
Mr Blake: Is your recollection that workarounds were routinely communicated to subpostmasters or only communicated to those who phoned the helpdesk with a problem?
Alan D’Alvarez: It depends on the workaround. So if the workaround is as part of your – what you do for your daily business, you have to apply this because the solution is not quite performing as we expect to it, that should be proactively communicated. If there’s an issue where there is perhaps a failure – you gave an example before an error message. You know, sometimes error messages could come up.
We wouldn’t necessarily – it’s not necessarily appropriate just to say to everyone, “Here’s 101 workarounds” or there’s ten workarounds or whatever the number is but that might be sporadic, comes up once or twice, therefore it’s appropriate for the service desk to have that as a known error, that if they do get a call they can say “Ah, it’s a known error, this is how you workaround that error. So it depends on the workaround.
Mr Blake: Still on this document, another topic – I’ll return to this topic but just while we’re on this document – can we just look at page 5, please. There’s reference there to the excessive number of recoveries and screen freezes.
If we go to the third entry, the third row down, it says, “Excessive number of Recoveries/Screen Freezes” and on 11 March that’s closed, and it says:
“Superseded by introduction of r108 at data centre which has significantly reduced the screen freezes. The data on recoveries is being separately progressed by IT and Gareth Jenkins …”
Do you recall this at all?
Alan D’Alvarez: I recall there were screen freezes that impacted the Post Office, and I can see that – I don’t recall specifically this. I was aware of screen freezes as an issue.
Mr Blake: How about recoveries? The recovery issue, that was a particular issue that was identified.
Alan D’Alvarez: I don’t recall that.
Mr Blake: Is it fair to say that it’s been closed but it hasn’t been eliminated because it says that it significantly reduced the screen freezes but it doesn’t say there are no longer any screen freezes and, in terms of the recoveries issue, that’s being progressed by Gareth Jenkins; it hasn’t been closed because that issue has been resolved?
Alan D’Alvarez: But it says “as per 146” so is 146 open? In which case, it’s been – that data recovery is if you go up –
Mr Blake: That’s been closed. That’s slightly further up on the same page.
Alan D’Alvarez: Now, we’ve got to go to 148.01.
Mr Blake: If we go over the page, that’s page 6, it’s about halfway down, it’s a new item there, 11 March.
Alan D’Alvarez: Okay. I don’t recall it specifically but it looks as though they’ve created an item, so there was a number of issues all in one, part of those issues are being addressed, others still remaining. So it looks to me, I don’t recall this specifically, but it looks to me they’ve opened the new action so not to confuse it with – the screen freezes may have led to other consequences and they want to track the other consequences.
Mr Blake: The screen freeze issue itself had been significantly reduced but not eliminated?
Alan D’Alvarez: I – again, I don’t recall –
Mr Blake: Can we go to the bottom of page 5. It says:
“As per Action 147.01 it was agreed that rollout would not be [recommended] until a period of stability had been achieved. GB/WR to agree what is meant by ‘Stability’ and what would be an adequate period.”
Then if we look in the column next to it on the page above, 11 March:
“A set of criteria has been provided by POL and this was reviewed for understanding in the meeting. Fujitsu will provide much supporting data/evidence by … 12th March to enable a decision to be made if pilot can restart”, et cetera.
“It was agreed that the data would not be perfect or complete, but needs to provide sufficient relevant information if a restart decision can be made.”
Again, in terms of the working relationship between POL and Fujitsu, it seems there that the criteria provided by the Post Office, in terms of resuming the rollout – were Post Office, at this stage – so we are 11 March 2010 now – the ones who were driving forward the rollout?
Alan D’Alvarez: Sorry, so when I read that, it’s – I think as I’ve explained before, when we take a decision that the solution is not – or the risks of moving forward further with our pilots and/or deployments is such that we’re going to freeze or have a halt or pause, the – you know, we would work jointly but the ultimate decision is with Post Office.
So here, what I read from that is that we’ve received a set of restart criteria. So with this issue, we needed make sure we understood what would be the basis of us understanding that it’s resolved and to give ourselves confidence because at the date – and we’re talking about stability, we’re talking about screen freezes, I can’t be sure, but this may be related to a significant issue we had with Oracle, an extremely significant issue we had with Oracle that did impact the offices intermittently that had migrated to HNG-X in as much as it loss – it didn’t lose connectivity, the connectivity hung and they could no longer – so the screen freezes – no longer trade until the branch database come up again.
And that was a very significant issue. So what we had to assure ourselves before we deployed further, once we’ve deployed fixes and things, that we understood what is the criteria for restarting, and make sure – and Post Office had the final say. So is it a period of “N” number of days, “N” weeks, whatever, is it acceptable to have? Because if you have a screen freeze, we had – in the Oracle issue that we had, there were regular, you know, throughout the week, four or five times, where all the post offices operating in the pilot would be impacted for a period of 15 minutes to 30 minutes, and that’s significant when they’re serving customers.
But, again, there could be a screen freeze for reasons not related to that. So you’re not going to say you had no screen freezes whatsoever, it could have been an intermittent one-off screen freeze. So the criteria was very important, that once you stop a deployment, to actually understand – do you understand what the issue is and, for us to make sure we make the right decision, do we have a clear set of criteria that we’re going to apply to the decision? Have we met that criteria to go forward?
Again, that criteria was also, dare I say it, to protect the joint programme from people wanting us to move forward. Come on, you’ve got to keep going. So we haven’t met this criteria, or we have met the criteria so we can demonstrate to ourselves and to our respective organisations the appropriate time when – why is it appropriate to move forward or why is it not appropriate to move forward?
Mr Blake: You mentioned earlier in your evidence about the risk versus progress balance.
Alan D’Alvarez: Yes.
Mr Blake: Were Fujitsu and the Post Office both very much aware that that is the balance that was in play at this time when there were those evident issues still cropping up?
Alan D’Alvarez: We made them aware. Because that’s our job. I mean our job as programme management is – that’s what we do. So it’s my job to make my organisation aware and Mark’s job to make his organisation aware of that.
Mr Blake: I’m going to return to the balance trading statement issue. Can we look at – do you need a break at all?
Alan D’Alvarez: I’m fine, thank you.
Mr Blake: Thank you.
Sir, just for your information – I’ll deal with the balance trading statement issue and then, in about – well, around 11.30 we may be able to take a break. I’m hoping that we will be able to take one longer break this morning, and not need to take a lunch break. That’s my intention. But my intention yesterday didn’t prove correct, so –
Sir Wyn Williams: I have every faith in you bringing home your intentions, Mr Blake.
Mr Blake: Thank you, sir.
Can we look at FUJ00094265, please. This is an email, you spoke earlier about raising the issue with Fujitsu’s general counsel and that’s Jean-Pierre Prevost; is that right?
Alan D’Alvarez: He is the legal representative assigned to Post Office.
Mr Blake: We have there an email from Geoff Butts saying:
“J-P
“Can you review this draft response [in] an email from Post Office … about Fujitsu’s approach to resolution and interim management of a software defect relating to the Final Balance Report for Branch Trading Statements, and let me know if any changes are required before it is sent. I’ve copied the HNG-X Leadership Team for information. This issue has been flagged as critical to fix before the start of rollout.”
In terms of those names at the top, are they all Fujitsu names?
Alan D’Alvarez: Yes.
Mr Blake: Then if we scroll down, these are the answers to the questions that have been requested by the Post Office, or proposed answers that are being run by general counsel. Were you involved in drafting this in any way?
Alan D’Alvarez: I would likely have reviewed it. But not specifically to –
Mr Blake: Do you remember reviewing it?
Alan D’Alvarez: I don’t recall reviewing it but it would be my normal operation to review it. Although, having said that, I may have reviewed it as part of being a CC list.
Mr Blake: If we scroll through it, it explains the cause, addresses the key questions, and it’s the final entry, really, that I want to read out now. We’ve seen this document before. It says:
“Can Fujitsu provide a commitment to support POL in proving the integrity of the system in any subsequent legal action (specifically where the difference in the two reports is used as a means to challenge the integrity of the system)?”
The proposed answer is:
“Yes, Fujitsu is willing to positive commitment to prove the integrity of the system in any subsequent legal action.”
We do, then, have the final version that was sent. Can we look at FUJ00142190, please. It seems to have been sent to Mark Burley by yourself on 8 April 2010. The email chain I just took you to was February, we’re now moving, so some time has passed. We’re in April. This says:
“Dear Mark,
“Please find enclosed a statement detailing Fujitsu’s position with respect to the Final Balance Report issue from the HNG-X pilot. This statement has now been confirmed by our Commercial, Legal and Programme Leads.”
It’s sent by you. Do you remember sending this letter?
Alan D’Alvarez: I don’t recall specifically sending it.
Mr Blake: Yesterday, Mr Burley couldn’t remember what happened to this issue. He couldn’t remember whether a letter was sent or not. We have here the letter. Do you remember confirming it with Commercial, Legal and Programme Leads at all?
Alan D’Alvarez: So it is likely that the advice from our counsel is it should come from me because I was the head of the programme or head of the transformation – lead of transformation. As I said, we would have – I would have reviewed that but we would be taking legal counsel specific to this statement. It’s out of my sphere of expertise. So I’m – I will be solely reliant on legal counsel’s view on that. I haven’t seen – you can – I don’t believe this was in my pack, this letter.
Mr Blake: It was in your pack.
Alan D’Alvarez: It was? Okay. Sorry.
Mr Blake: It certainly has been provided to you or it may have – forgive me, if I’m wrong on that, but in any event, it’s got your name on it.
Alan D’Alvarez: These kind of – I’m stretching my memory. I do remember the – having reviewed the content because it wasn’t just the legal – I don’t know whether this letter contains just the legal statement or whether it also has an explanation of all the questions or response to all the questions that was posed by Phil.
Mr Blake: Let’s turn over the page, then.
Alan D’Alvarez: It does. So it’s, basically – yes.
Mr Blake: What’s interesting in this letter is if we turn over the page, the final entry here is different: only slightly different, but significantly different, in its content. Can we just blow up that final section and slightly above as well, from “Can Fujitsu”.
It says there:
“Can Fujitsu provide a commitment to support POL in proving the integrity of the system in any subsequent legal action (specifically where the two reports is used as a means to challenge the integrity of the system)?”
That’s the question posed and the answer is now:
“Fujitsu is willing to provide a commitment to assist in trying to prove the integrity of the system in any subsequent legal action in accordance with the existing contractual arrangements.”
Do you remember that form of words being inserted?
Alan D’Alvarez: I don’t recall it, you know, but I can read that and that would have been on the recommendation of legal counsel.
Mr Blake: The words, before they were changed here, are both interesting because they say “Fujitsu is willing to provide a commitment to assist in trying to prove the integrity of the system”. So rather than supporting POL improving the integrity of the system, they are providing a commitment to assist in trying to prove the integrity of the system in any subsequent legal action. Was Fujitsu at this stage concerned that they couldn’t actually prove the integrity of the system in any legal action or they might not be able to prove the integrity of the system in any subsequent legal action?
Alan D’Alvarez: Not to my knowledge.
Mr Blake: Would you have been happy at this stage to have put your name to a document that said that you are willing to prove the integrity of the system in any subsequent legal action?
Alan D’Alvarez: So, firstly, if you go back to the original draft –
Mr Blake: That’s FUJ00094265. It’s the same wording as the beginning of that sentence in that original draft. It’s page 2.
Alan D’Alvarez: Okay, and it’s specific –
Mr Blake: You can – perhaps we can bring them –
Alan D’Alvarez: No, that’s fine.
Mr Blake: Can we bring those side by side? So it’s page 2 of this document and page 2 of the document that ends 190, and perhaps we can highlight the “Can Fujitsu” section the final section on that page, and the final section on the other page.
Alan D’Alvarez: It’s actually the question, so if the question is the same I should have read that. So the question is specific to the two reports. When I was reviewing and the briefings that I got with regards to this specific issue, it was very clear that the underlying data that was held on the system was correct and it was how we – how we presented the reports or how the reports were – got their data was where the issue was. So it wasn’t the data on the system, that had integrity issues, it was how the report was created, it was an issue where it was going from a step in the process where it could have had data overwritten and therefore, you know, not had the correct data from the system.
So the integrity of the solution on this issue was not in question. It was – the report was taking its information from an incorrect area of the system, which has volatile memory and therefore open to change, and the fix was to make sure it went straight back to the source information on the branch database.
So the integrity of the system specific to this, there was no – from my perspective, there was no issue with the integrity of the system. I was satisfied when I – David John’s was my technical lead and I had extreme confidence and he was satisfied and he explained it to me and I was satisfied.
The question below seems to be a wider – it appears to me just a wider statement of any evidence we might give at any time for any reason.
Mr Blake: Absolutely. Actually, I mean, forgive me, I made a mistake. The wording is actually different between those two versions if we look at them now side by side. It’s even more caveatted in that one on the right-hand side and, as you say, it’s a question about can you prove the integrity of the system. So we can put aside the branch trading statement issue, and this addresses really the system as a whole.
The final wording – the initial wording is we’re willing to provide commitment to prove the integrity of the system, “a commitment to prove”, and now it’s not only changed at the beginning – so it’s “a commitment to assist in trying to prove the integrity of the system” – but then also there’s that bit about existing contractual arrangements.
Now, forgive me if this wasn’t a document that was provided to you in your pack. That was would be my fault but looking at it now and considering it, does it bring back any memories of this particular issue?
Alan D’Alvarez: Not particularly but that first one would be from our investigations, from our understanding, our proposal. That would be the programme’s proposal of how confident we were and comfortable with regard to this. But we take legal counsel advice. So what comes back I’m not going to dispute because they are the experts in this matter.
Mr Blake: Does it suggest that, at this point in time, Fujitsu didn’t have sufficient confidence itself that it could prove the text of the system?
Alan D’Alvarez: I don’t think it suggests – the way I read that is any computer system may develop a fault which may, at any time, have some impact on integrity, as we had the one at Derby. And when we had that issue at Derby, I – under – you know, I initiated through my line management a review of the solution.
So – because we were concerned that there were potential – and they gave us confidence what happened at Derby and why it happened we didn’t pick up in testing, we understood that, and they also looked at the whole system and the protections we put in.
Now, there’s – there may always be a change that’s made on the system or peculiar circumstances of, you know, events that happen on a particular machine, which may create a condition. So you can never absolutely say all the time but, again, what we should be doing is supporting by saying this is what we’ve done to demonstrate the integrity of the system and at the point of whatever investigation you are on, were there any known incidents raised that may or may not have an impact on that.
But that’s, you know, what you’d expect to happen in any investigation. They wouldn’t just say at a point in time, back in when – I think it was February we done that report, because Derby was at the end of January, wasn’t it, and it was very quickly turned around because it was so urgent.
So, at that point in time, but it doesn’t mean it wouldn’t change in the future if a release goes out potentially as an intended consequence.
Mr Blake: I’m going to take you to two documents before we take that break. The first is FUJ00142193. Forgive me, this may again be one of the documents that wasn’t in your pack. I think that’s the last of the documents that wasn’t in your pack, and these are ones I think have only gone through our system relatively recently.
Please do say if you need more time and I can provide this to you over the break if you want to re-read it as well.
If we could look at page 3. We’ll start by looking at page 3. We’re now at 29 March, and your letter was 8 April, so just before – so this is before your letter was sent.
One of the requests from Phil Norton, looking down that page, the final paragraph there is they are proposing a form of words:
“Can Fujitsu provide a commitment to support POL in proving the integrity of the system in any subsequent legal action (specifically where the difference in the two reports is used as a means to challenge the integrity of the system)?
“Yes, Fujitsu is willing to provide commitment to assist in trying to prove the integrity of the system in any subsequent legal action.”
POL Legal are proposing there a different form of words:
“Fujitsu is convinced of the integrity of the HNG-X system and as such will, at its own expense, provide a commitment to POL to assist in trying to prove the integrity of the system in any subsequent legal action.”
So it seems there POL Legal is proposing a statement that says that Fujitsu is convinced of the integrity of the system. Is that something that you remember?
Alan D’Alvarez: Jogged my memory in getting a response. I do remember immediately – now I’ve seen this, I do recall receiving this and discussing this with Gavin Bounds, who is the Business Unit Director, and agreeing that this is now both a legal and a commercial question being given to Fujitsu.
So again, I sent it to legal counsel, but we also looked at commercially what was our commercial position. But again, I would take their advice from this point onwards.
Mr Blake: Would it be typical for the Post Office to provide, and Post Office Legal to propose, forms of words to be adopted by Fujitsu?
Alan D’Alvarez: This will be the first time I’ve seen it.
Mr Blake: If we go to the first page of this email. From Phil Norton, 8 April. So this is a day – the same day as your letter is dated, and it’s sent to you. I’ll just read that out. It says:
“Alan
“As discussed at the interim JSB, please be advised that POL do not accept the latest Fujitsu response. The area that Fujitsu need to reconsider concerns the proposal from POL Legal that Fujitsu will, at its own expense, provide a commitment to POL to assist in trying to prove the integrity of the system in any subsequent legal action. This was not evident in the proposed final version of the statement.”
So there was a version that was proposed where that form of words was not included.
“In the statement from Fujitsu they seek to limit the cost of their involvement in any subsequent legal proceedings to be in accordance with the existing contractual arrangements.
“As the error is entirely of Fujitsu’s making they should not seek to limit their assistance as per the contract. Would you please arrange for the statement to be revisited and a more agreeable form of words to be proposed.”
Do you remember this?
Alan D’Alvarez: Not directly, but I do remember that, you know, there was further conversations with regard to a proposed legal statement and, as I said, that then passed it to my direct report on the account, and we agreed that it needs to go to Legal and Commercial for a response. But I cannot recall what that response was.
Mr Blake: Do you recall if the letter was changed in any way, or the statement?
Alan D’Alvarez: I cannot recall.
Mr Blake: They’re quite strong words from Phil Norton there.
Alan D’Alvarez: Yes.
Mr Blake: Did you get the feeling that you were being dictated to a bit by POL in relation to this issue?
Alan D’Alvarez: I – I mean, looking at this, it’s a thing that we often have with customers, you know, who is going to bear the liability of any costs going forward and, again, that becomes a contractual position. Again, you know, it’s – when I saw the previous response, it occurs to me that – you know, it comes to cross to me that there was a specific question asked and the response was for a more general thing.
So whether the discussion – I do remember briefing our legal department as to we had a review of the integrity of the solution, we got some additional reactions just to kind of assure ourselves with regard to strengthening our testing and things going forward.
But in any discussion I’ve have on any system, I say that at a point in time, you can review and it’s fine but it could always be a work where something, you know, causes a change which could compromise that and we’ve always got to be alert to that and that’s why we have support.
Mr Blake: Thank you, if we just scroll up on this email and this is my last question. Are you able to assist us with the recipients of this email? I see Gareth Jenkins is a recipient. Are you aware of him being aware of this particular issue?
Alan D’Alvarez: Well, clearly because he’s – I took a – when Geoff was off, I was pointed in his direction that Geoff was working with him and others with regard to the issues around the branch trading statement. So, clearly, I know that Gareth was aware and Gareth did respond to Phil Norton. So Phil would know that he was engaged so it was right for him to be on this.
Mr Blake: Would Gareth have been aware that Fujitsu didn’t want to provide that blanket assurance?
Alan D’Alvarez: He’s on this email. So he has got the opportunity to be. I didn’t discuss it with him directly but he’s on this email.
Mr Blake: Can you give us an indication of who these other recipients are?
Alan D’Alvarez: Okay, so Debbie Richardson is the head of testing, Geoff Butts was the deployment lead, Graham Allen was my head of development, Graham Welsh was the lead customer service representative, which we had engaged in our Hypercare. Jean-Philippe Prenovost was our legal counsel. John Wheeler, Mark Andrews and Kevin Talbot I believe – I can’t say – I believe they’re Post Office. I don’t believe they’re Fujitsu. Mark Burley is Post Office, Will Russell was Geoff Butts’ opposite number in deployment, and Mark Burley was my opposite number.
Mr Blake: Thank you very much I think that’s an appropriate time to take our break.
Sir, I will conclude – if we take a 20-minute break now, I imagine I will be done within an hour or so.
Sir Wyn Williams: All right. Just to let you know that I may literally have to take a three or four-minute break at around 1.00, as it happens, Mr Blake, just to take a phone call. But I’m sure we will manage it some way or another, which is reasonable in all the circumstances, all right?
Mr Blake: Absolutely. Unless people get particularly hungry we could take two breaks this morning and achieve it that way.
Sir Wyn Williams: Well, whatever. As we get closer to 1.00, we will review precisely what we are going to do. All right.
Mr Blake: Thank you very much.
Sir Wyn Williams: Thank you, Mr Blake. So 20 minutes from now.
Mr Blake: Thank you.
Mr D’Alvarez remains obviously on oath and won’t be speaking to anybody about his evidence.
Sir Wyn Williams: Fine, thank you.
(11.35 am)
(A short break)
(11.57 am)
Mr Blake: Thank you, sir.
Mr D’Alvarez, can I ask you to look at FUJ00094296, please. This is a document from around the same time, 3 March 2010, and you’ll see in the top right-hand corner your name as a recipient. There is reference in this email chain to a fix to get data for a court case, and I just wanted to know if this was a similar issue, the same issue or something totally different.
Let’s look at page 3, the bottom of page 2 and top of page 3. We that have an email there from Pat Lywood, who is the service implementation manager, and she says:
“I will cross-check these with prayers [that’s something relating to the below]. However I suspect, for today at least, that the fix to get data for the court case may be more important than these fixes.”
If we go to page 2, so above that, you say:
“Pat,
“What is the fix for the court case?”
Then if we go above that one and Graham Allen says:
“A fix to Audit – was developed yesterday and passed through Integration last night.”
Do you recall what this exchange is about at all?
Alan D’Alvarez: I recall there was an incident raised where they – with regard to the audit server having duplicate – so, basically, having duplicate records stored to it and when reports were – the previous system would extract any duplicates and just have the single version of what happened, and the new version of what we implemented did not have that duplicate spotting extraction purpose – functionality.
So I assume that that is with regard to – something with regard to the information stored on the audit server.
Mr Blake: So this is a totally separate issue to the branch trading statement?
Alan D’Alvarez: I believe so, yes.
Mr Blake: If we look at page 3, that email, the first email that I showed you, it refers to a fix to get data for the court case. Can you assist us with what that means? It’s the top of page 3, thank you.
Alan D’Alvarez: No, I can’t. That’s why I asked Pat, question mark, “What do you mean, fix for a court case?”
Mr Blake: Did you ultimately understand, having asked that question, what that meant?
Alan D’Alvarez: She replied saying it’s the audit fix, so I probably didn’t pursue it more other than made an assumption that they were looking to retrieve audit records and they wanted that fix in there, so that the audit records that they retrieved was accurate.
Mr Blake: So does it suggest there that there was an issue with the accuracy of audit data at that time?
Alan D’Alvarez: Not accuracy. It’s just that it would – there would be potentially duplicate records which you would have to filter out.
Mr Blake: How do you know that it’s a duplicate issue rather than a reliability issue?
Alan D’Alvarez: I’m only – I’m presupposing that we had an issue with audit records and duplicate audit records, and we had to provide the fix for it.
Mr Blake: You remember there was a specific issue with duplication?
Alan D’Alvarez: I remember there was a specific issue with the audit and I’m just making an assumption that that is to do with that.
Mr Blake: Similar period but a different issue and I think it was one you were talking about earlier, can we look at FUJ00093030 and this the “HNG-X Counter Review” issue. If we look at that bottom email, it’s an email from Stuart Rye, who was Stuart Rye?
Alan D’Alvarez: He’s, I believe he’s a business consultant, and he was brought in as part of, I think he was brought in by – let’s have look at the names, David Leask. David Leask, he’s the chief technical person alongside Maz who is the chief programme person for private sector and they bought in two people to undertake an independent review.
Mr Blake: This is exactly that issue you were talking about earlier in your evidence?
Alan D’Alvarez: Yes.
Mr Blake: Is that other person Paul Roberts, who was copied in there?
Alan D’Alvarez: Yes.
Mr Blake: So they were two interpreter individuals who were asked to conduct a review –
Alan D’Alvarez: Correct.
Mr Blake: – by Fujitsu –
Alan D’Alvarez: Correct.
Mr Blake: – because of a concern that Fujitsu had about, it says there, “the incident of a duplicated basket at the end of January”; do you remember that issue?
Alan D’Alvarez: That is the Derby – that is the Derby incident which I immediately engaged with Maz that I wanted to assure, because it is a serious incident and we need to understand (i) the integrity of the solution and (ii) how we missed it in testing.
Mr Blake: Thank you. If we scroll up, that email and that report, the review – it’s volume 1 there, 9 February 2010 – is sent to Gareth Jenkins by Graham Allen. Do you know by Gareth Jenkins would have been asked to comment on that report?
Alan D’Alvarez: I believe Gareth was the one that was assisting Geoff in the analysis of – oh no, sorry this one? I assume because he is the recognised counter SME, the subject matter expert, for Horizon?
Mr Blake: So issues – a concern that you had internally in Fujitsu with regards to the integrity of data because of a duplicated basket, that report was being sent to Gareth Jenkins because he was the lead on what issue?
Alan D’Alvarez: No, he’s the SME. He’s the subject matter expert that – whether he was still working in the customer services area or whether he was part of the architecture group, we have subject matter experts, so he’s the recognised a counter application subject matter expert.
Mr Blake: I’d like to look at that attachment and that can be found at FUJ00093031, please. This is version 1 of the report. I know you’ve been handed a version 2 and we’ll get to that and I’ll give you time to look at that. But this is version 1., and that’s, it seems, the version that was attached to that email.
Now, the circulation, is that an internal circulation list; is that right?
Alan D’Alvarez: At this stage, yes.
Mr Blake: Then you have there the two authors, the independent authors. I’m going to read to you. If we scroll down, it explains what the problem is there. It says:
“On 28th January 2010, the Data Reconciliation Service … process detected an error in a banking transaction. Subsequent investigations revealed that the Branch database had two transactions with different JSNs but the same SSN for a specific Counter on that day but the 3rd Party banking system only had one transaction. The clerk did not know that a duplicate transaction had been created.”
So as you’ve said, that is self-evidently a serious issue?
Alan D’Alvarez: Yes.
Mr Blake: It then goes on to say:
“An analysis of the database has revealed one other occurrence, again at Derby but on a different day and involving a different clerk.”
So is that the same Post Office but it’s a different individual?
Alan D’Alvarez: That’s what it says, yes.
Mr Blake: “The net effect would be that the Post Office and Branch records would not match. Where this happens, the Post Office investigates the branch and postmaster, with a view to retraining or even uncovering fraud. It would seriously undermine Post Office credibility and possibly historic cases if it could be shown that a discrepancy could be caused by a system error rather than a postmaster/clerk action. Most importantly, the central database as the system of recorded would be called into question.”
Now, at this stage, whose words was that? Was that the two independent authors who had written that?
Alan D’Alvarez: Yes.
Mr Blake: Can we go over the page, please. The second paragraph there. It says:
“The development team concluded the failure was caused by a bug and a resolution has been identified which includes further measures to remove the possibility of this occurring in future.”
So there’s no dispute here, this was very much caused by a bug in Horizon Online.
Alan D’Alvarez: Yes.
Mr Blake: If we go over to page 5, there are the conclusions. Thank you. It says there in bold:
“Overall, the actions taken to redress the Derby issue are appropriate. We believe the Counter Application fully supports the need to protect the integrity of financial transactions.”
What do you understand that statement to mean?
Alan D’Alvarez: It means that on the request I didn’t want them to just look at this specific incident because we’ve already diagnosed the specific incident and got a conclusion. I wanted them to look at how we’ve implemented the counter application to see whether there’s any flaws in its design or its implementation.
Mr Blake: My reading of that is that it isn’t saying that it guarantees the integrity of all financial transactions or anything like that, it’s using quite careful language that the counter application fully supports the need to protect the integrity. So it’s identifying the need to protect the integrity of financial transactions rather than signing off the integrity?
Alan D’Alvarez: I took that as, from what we’ve designed and implemented, it fulfils a brief that we have – should have high levels of confidence in the integrity of the solution. I think it gives examples as well as to how we put protections in.
Mr Blake: Sorry, you’ve used the word “solution”. Did you intentionally mean “solution” rather than “system”?
Alan D’Alvarez: The – okay, the system. But the counter application solution is what I was looking at. So the solution – okay. I would, in this context, one and the same system/solution, the solution being we have a counter application. Looking at the counter application itself, is it defensive against system failures? So the system can fail at many different ways, power cuts, cables being, you know, intermittent, kind of – all sorts of things could cause a system failure. Is the solution robust enough to manage those error conditions in a way which doesn’t impact integrity? So I do mean the solution, as in the application. Is it defensive?
Mr Blake: I’m happy for you to take a little bit of time just reading that to yourself, that passage in bold, just to be absolutely sure that that is a fair reading of what is said there.
Alan D’Alvarez: Yes.
Mr Blake: So it’s your reading of that that actually that is saying that the counter application has integrity –
Alan D’Alvarez: Yes.
Mr Blake: – not that it supports the need to protect the integrity or – did you not see a distinction between those two?
Alan D’Alvarez: I haven’t read it that way, personally.
Mr Blake: We saw earlier that there was a request from the Post Office to provide a commitment to support POL in proving the integrity of the system, of – yes, of the system. Was this, do you think, meeting that earlier request in any way?
Alan D’Alvarez: I think it was a later request.
Mr Blake: Sorry, the later request. Do you think – you were asked for a statement to sign off the integrity of the system. Looking at this, does this look like a statement to the effect that it is signing off the integrity of the system?
Alan D’Alvarez: In the previous example, which I believe is the Warwick example, the question posed was, could we sign off the integrity of our solution with regard to the branch trading statement? That specific question. And, absolutely, this is looking at the application and is it defensive? I would like to refer to – I’ve said a number of times now that, from our position, we’ve satisfied ourselves that what we’ve implemented will demonstrate and actually fulfils an integrity brief but things can happen on any system which could change that, so that’s a static point in time.
Mr Blake: So it’s a point in time that addresses a specific problem that this report is addressing?
Alan D’Alvarez: And also a request for me to have a design and code review of what was implemented to say that – is that appropriate and robust with regards to normal system standards for integrity of financial data.
Mr Blake: Because when we looked at the earlier example, you drew a very clear distinction between signing off the integrity vis à vis the specific issue and signing off the integrity of the system. Here, you’re saying that the – your reading of this report is that, at this point in time, it was signing off the integrity of the system?
Alan D’Alvarez: Of the application.
Mr Blake: The application being the counter application?
Alan D’Alvarez: The counter application, the solution for the – what we provided to Post Office for the postmasters and their staff to actually interoperate with Horizon. So this is the integrity of the counter application.
Mr Blake: But the counter application must include the cash account, for example?
Alan D’Alvarez: Yes, it does.
Mr Blake: So it would be signing off the integrity of the cash account?
Alan D’Alvarez: Yes.
Mr Blake: Do you think that was the author’s intention in this passage?
Alan D’Alvarez: I’ve concluded – my – I believe so, yes.
Mr Blake: Can we turn over the page, please. We’re still on conclusions. 3.20, there is a conclusion that reads as follows:
“Stock check is not a reliable method for catching issues due to the erratic nature of the stock check occurring.”
Is that something that you recall or understand?
Alan D’Alvarez: Not specifically but stock check, from my understanding, is just a check of what the system says it’s – of stock.
Mr Blake: Why might it not be liable?
Alan D’Alvarez: I can’t recall.
Mr Blake: Is that something that occurs on the Horizon System, the stock check?
Alan D’Alvarez: I believe a postmaster can do a stock check at any time.
Mr Blake: It’s not a physical stock check of how many stamps are held in branch; it’s something that is using the Horizon system to compute?
Alan D’Alvarez: Yes.
Mr Blake: If we turn over the page to “Recommendations”, we have recommendation – I’m going to take you to 4.5 and 4.6. 4.5 is:
“Consider advising the Post Office of the benefit of more effective stock control as an indicator of clerk errors or Fraud.”
What do you understand that to mean?
Alan D’Alvarez: I don’t at this stage recall what that refers to.
Mr Blake: I mean, reading it now, what does it seem to you to mean?
Alan D’Alvarez: It’s given – it’s just highlighting that the system may allow variances to be input into the system.
Mr Blake: Does it suggest that there should be another method in order to assure the Post Office or assure yourselves that there is, in fact, either a clerk error or fraud?
Alan D’Alvarez: That’s what it says but I don’t recall specifically back to the intent of that, but that’s what it suggests.
Mr Blake: Would you advise the Post Office of ways in which to improve the reliability of their prosecutions for fraud, for example?
Alan D’Alvarez: I would not but if it felt that Fujitsu felt it was appropriate to at least have that conversation then I would pass that advice on.
Mr Blake: So it’s not that that recommendation itself is problematic because it’s suggesting that you advise the Post Office of something?
Alan D’Alvarez: I don’t believe so, no.
Mr Blake: 4.6:
“Review and strengthen negative testing, if appropriate. The recent problems reflect the asynchronous nature of the new application and traditional or historic test cases may not reflect this.”
Can you tell us, what is negative testing?
Alan D’Alvarez: So negative testing, so typically when you test a system you run a set of tests which demonstrate that if you go through the right processes and that and everything is set up properly the system performs as it should.
Negative testing will include things like what happens if you put in the wrong inputs? What happens if a cable is disconnected? What happens if it loses connection with the data centre? So we would deliberately create scenarios which is not normal operating scenarios, to see whether the system recovers gracefully from those situations.
Mr Blake: So you might have some testing, is it positive testing; is that the opposite of negative testing?
Alan D’Alvarez: Yes.
Mr Blake: Yes, positive testing, where you’re testing a case where a subpostmaster puts in exactly the right information, presses exactly the right keys that you’re meant to do?
Alan D’Alvarez: Following the processes, yes.
Mr Blake: That’s positive testing. Negative testing is what if the subpostmaster presses the wrong key at a particular time?
Alan D’Alvarez: Correct – or what if it goes offline? What if a cable gets pulled out?
Mr Blake: Absolutely. One of the recommendations here is there should be more negative testing?
Alan D’Alvarez: It’s to review. Have we created all the scenarios? Because I believe one of the trigger points for this was a capacity constraint that was caused by, I believe – because we were monitoring the system all the time during pilot, we were picture putting diagnostics and performance monitoring things to make sure everything was going right and it was believed that that created a capacity issue, which then created the issue which led to the incident.
And, in our testing, we was looking at normal as if it was rolled out. We did not take account of, during the pilot, which is a test phase, we’re going to put additional monitoring tools into the thing which created more usage, so we didn’t stress the system the way.
So that’s something we did miss specifically and that’s why we missed this in our testing, because we assumed it was – those tools that we put in to support the pilot were only there temporarily, and that wasn’t taken account in the testing.
Mr Blake: Thinking back to this morning, the very early questions that I was asking you about, and I took you to that document that I discussed with Mr Burley yesterday, where there is a suggestion of reduction in the testing that occurred before the pilot took place, and I think your evidence was “We didn’t cut down on the testing, we just did it while the pilot was taking place”.
I mean, looking at that combined with this 4.6, reflecting on it, do you think that there was insufficient negative testing before pilot?
Alan D’Alvarez: Performance testing isn’t negative testing. That’s non-functional testing.
Mr Blake: Yes.
Alan D’Alvarez: Negative testing is creating a failure condition. So the volume testing is specific to the data centre and how much loads can the data centre take? So it’s completely different from this scenario.
Mr Blake: Putting aside volume testing, on reflection, and looking at this recommendation, do you think there was sufficient or insufficient negative testing before the pilot took place?
Alan D’Alvarez: Clearly, this showed that consideration wasn’t taken to some of the additional tools that we were using to monitor the solution during the pilot. So it found a weakness in our testing that we had to review and we filled that.
Mr Blake: In plain English, do you think there should have been more negative testing before the pilot?
Alan D’Alvarez: There was a miss in the scoping of our negative testing. When you say more, we should have – there should have been consideration of the additional tools that was temporarily in place in the pilot, and that was a miss in the test analysis.
Mr Blake: Forgive me, “a miss in the test analysis”, for me, maybe just me, is not plain English. In plain English, should there have been negative testing before the pilot?
Alan D’Alvarez: There was negative testing before the pilot, but it didn’t capture this scenario.
Mr Blake: So was there sufficient negative testing before the pilot?
Alan D’Alvarez: When the analysis was done we believe so but this in hindsight shows that there was a miss.
So in hindsight, on the negative set of testing, you could – it starts to get exponential as to how many different failure conditions you could create, and it comes to – I’ve seen negative testing where you actually question how could you have that scenario where this happens at that time and that time? So it’s a case of you take – you know, a take a balance as to what are the likely scenarios? Then you also have some good test resource, and we had some very good test resource, that would just play about with the system and just do what-ifs. But at the time the analysis was done and it was believed and, you know, it was jointly reviewed with us in the Post Office, that the testing in the negative side was sufficient but this showed that it could have been more robust. But that’s in the benefit of hindsight.
Mr Blake: We know what subsequently happened, and events that have affected people’s lives. Looking at what ultimately happened, do you consider that there was sufficient or insufficient negative testing?
Alan D’Alvarez: So I considered at the time our testing was sufficient. The pilot phase is also a test phase. So this was picked up in a test phase. It’s a live test. So the pilot phase picked up a scenario that we didn’t have in our laboratory testing and picked up this error before it went to general, and it picked it up very, very quickly, with just the 10 or 12 post offices that was part of the pilot.
Mr Blake: 4.6 isn’t just talking about this specific scenario. This is talking about negative testing more broadly, isn’t it?
Alan D’Alvarez: So they said review – so once – so it identified that in our laboratory testing, we did not cater for this. So the recommendation is, “Could you review everything you’ve done to see if you can think of other things you may wish to test?” And that action was given to Debbie Richardson to do so.
Mr Blake: After receiving the ultimate recommendations, and we’ll look at the final report, was there a significant increase or improvement in, and strengthening of, negative testing?
Alan D’Alvarez: I’m aware that there was areas of improvement, the size and number I would have to refer back to Debbie Richardson, and Lee Farman would know as well because they would have reviewed it jointly.
Mr Blake: So you’re not aware of whether there was or wasn’t a significant improvement in negative testing?
Alan D’Alvarez: I’m aware that they reviewed and they strengthened testing. How much I can’t recall.
Mr Blake: Can we go to FUJ00094290, please. This is an email from yourself to the general counsel, Jean-Philippe Prenovost, and it is the 2 March 2010. You’ve attached the “HNG-X Counter Review volume 2” that’s dated 24 February 2010, and you say as follows – it’s entitled “Integrity review of HNG-X – request for review”:
“Jean-Philippe,
“You may be aware that an incident was raised during the HNG-X pilot that there have been two instances where a single transaction was recorded twice on the system. As you can appreciate from your workaround PCI Compliance, the integrity of the system is critical to be able to support Post Office in any litigation they choose to take against Postmasters should they uncover any financial irregularities. As such, we took this issue very seriously and part of our rectification plan was sponsor an independent review. The report from this review is attached and it is my intention to share this with Post Office. Prior to sending, could I ask you to assure that there are no legal issues or consequences associated with the content of the review that need to be considered.”
What did you have in mind when you were asking Jean-Philippe Prenovost to review that report? Were you looking for comments on the content of the report and track changes and things like that?
Alan D’Alvarez: I was – it’s hard to recall my thoughts at the time but I would expect to be asking does this – you know, we have an obligation, we have a contractual obligation to support this. There’s a major issue, a major incident that occurred, which obviously has a material impact on that obligation. Here’s a report to show what we got. So I’m asking him to – from a legal standpoint, are we satisfied with this report that we can still fulfil our obligations?
Mr Blake: So are you satisfied with – were you expecting legal advice in relation to the overall report, rather than comments on individual entries and passages?
Alan D’Alvarez: Yeah. I wouldn’t, you know – and also, is anything in there something which we need to strength then or it’s incorrect from a legal perspective? So I’d expect Legal to assure themselves and to be aware themselves of this issue, and how we’ve addressed this issue.
Mr Blake: It certainly seems, at this time, you were very live to issues about the integrity of the system being critical to be able to support the Post Office in litigation. I mean, you’re raising that issue with him. Was that very much live in your mind at the time?
Alan D’Alvarez: Absolutely. It was the number 1 priority that was given to me by Mark Burley.
Mr Blake: Do you recall, say, in 2009, the Computer Weekly article about Horizon and concerns that were being generated around this period about the Horizon System?
Alan D’Alvarez: I don’t recall, no. I – sorry. My attention has been drawn to it now. I don’t recall knowing of it at that time.
Mr Blake: Was there anything in particular that highlighted this particular issue to your mind to raise it as such a significant issue?
Alan D’Alvarez: Reading that Computer Weekly, which I done probably before Christmas because it was part of the Inquiry discussions, that referred to Horizon, not HNG-X.
So when I took on the position to lead the programme, the first thing I done, as I would with any programme, was to sit with the customer and say “What your business priorities? What are the priorities? What are your business drivers? What are your drivers?” And Mark made it very clear to me, number 1, integrity of the solution, whatever we put out, we must ensure integrity. And, for me, that was sacrosanct.
Mr Blake: Can we look at FUJ00094392. It’s the bottom email of the first page that I’d like to look at. Thank you. This is 10 March now. You have copied below this email the feedback that you have received. So there’s been a review of the version that you sent to the legal team, and this is the feedback. I’m just going to read that feedback, if you could scroll down slightly – thank you very much:
First paragraph, page 2 – The following section is potentially problematic …”
That’s the section that said:
“‘It would seriously undermine the Post Office credibility and possibly historic cases if it could be shown that a discrepancy could be caused by a system error rather than a postmaster/clerk [error]. Most importantly, the central database as the system of record would be called into question’. As discussed, there is no need to paint this in the worst possible light. I would suggest the following as being accurate without being unduly alarmist: ‘If it could be shown that a discrepancy could be caused by a system error rather than a postmaster/clerk action, it could potentially call into question the effectiveness of the central database as a system of record’.”
It says there “further to our conversation”, can you recall the conversation or “further to our discussions” on Friday; do you recall the discussions about that particular paragraph?
Alan D’Alvarez: I recall a meeting with Legal, just to – because they wanted to understand better the scenarios and the context and everything else around it, not just “You’re giving me one problem”. So I remember a meeting. It was, I believe, about an hour long, and lots of things were discussed at that meeting. I also, I do recall them now, now seeing that, I do recall them asking – I think, as you did – who wrote that, and I said that was written – that wasn’t any words from the programme, that was written by the reviewers.
Mr Blake: So it was the words of independent reviewers. It was also very much highlighted in your covering email to the general counsel when you sent the original draft about the importance of the integrity of the system for legal proceedings. So it was something that was very much in your mind as well, and that was your evidence a few moments ago.
Alan D’Alvarez: Yes.
Mr Blake: Were you surprised to have received that feedback from general counsel?
Alan D’Alvarez: When you say surprised, that’s what our legal representative wished us to do.
Mr Blake: I mean, painting something in the worst possible light, that’s not necessarily legal advice, is it? That’s more public relations, isn’t it?
Alan D’Alvarez: All I remember is, in the discussions, being asked where did that come from? And I said it just come from – and it’s why Legal felt that we should be factual, that the – that it would be, you know, this issue, if not resolved properly, and we reviewed the rest of the system, the record of accounts would be done, is a better one. You’d need to ask Legal why their advice was that. But I would not go against our legal advice.
Mr Blake: You wouldn’t go against their legal advice but there’s nothing in that italicised section that’s untrue, is there?
Alan D’Alvarez: So when we say, “possibly historic cases”, I’m not aware of historic cases. I don’t know whether or how – whether that is just general discussion points that they’ve picked up during the review or whether they had evidence of that. I don’t know.
Mr Blake: The concern about undermining the integrity of the system, the central database as a system of record would be called into question, that was very much in line with your concern and the very purpose of requesting a report, wasn’t it?
Alan D’Alvarez: So my concern is the integrity of the solution, and does it (1) protect Post Office and their employees and (2) does it fulfil our contractual requirements? Historic cases has nothing to do with HNG-X so why that’s in there, I don’t know. What relevance that has to HNG-X, I don’t know.
Mr Blake: So we can take out the words “possibly historic cases” although it says “possibly” but let’s take that out. The rest of it, it’s accurate, isn’t it?
Alan D’Alvarez: It could be called into question if not resolved.
Mr Blake: Let’s look at what ultimately happened and the prosecution of subpostmasters and the overturning of those convictions by the Court of Appeal because of the reliability of the data that was being used in those prosecutions. I mean that was spot on, wasn’t it, that section in italics?
Alan D’Alvarez: I – I don’t have an opinion on that. It’s a case, this was a report, looking at HNG-X.
Mr Blake: But you could reflect on what’s ultimately happened. Look at those words in italics and agree or disagree whether that actually reflects the very reason why we’re here?
Alan D’Alvarez: As you put the question to me, yes.
Mr Blake: Let’s look at FUJ00142175 – sorry, can we stay with that for one second. Point 2:
“Section 3.19 on page 7 – You’ll recall that we cannot really make sense of what was being referred to here.”
I’ve looked back and it locks as though, and you can – we’ll go through it – but it looks as though that was actually 3.20 and that was one of the conclusions about the stock check issue that we just discussed.
Alan D’Alvarez: Mm.
Mr Blake: We can have a look at that and see that that also comes out. Thank you. We can go back to 175.
Can I ask, 175, so FUJ00142175, this is what we’re told is the final version. It has version 2.0 at the bottom right-hand corner but we are told it is the final version, version 3. Can we put up on the screen, I’m going to ask for a bit of work on this display once again, I’m afraid. It’s FUJ00093031. This is the earlier version I took you to, just so that we can look at some of the changes that happened after that feedback had been received.
Thank you very much. So on the left-hand side we have version 1, on the right-hand side we have version 2 – or version 3, actually, is the one on the right-hand side. Now, an initial difference we see there is that on the – it’s initially you’re one of the circulation list and that’s changed to one of the reviewers. Do you know why that change was made?
Alan D’Alvarez: Because they’re asking for a review, or they, sorry, they –
Mr Blake: It’s ultimately the same document.
Alan D’Alvarez: Yes, sorry. So it is a final report to say that those people that was on the circulation list had been given the opportunity to review and feed back comments.
Mr Blake: Because this is the version that’s going to go to the Post Office, you’re named as a reviewer rather than part of the circulation list; is that right?
Alan D’Alvarez: Correct.
Mr Blake: Yes, thank you.
Then if we look at the third substantive paragraph, that has been added on the right-hand side. It says:
“This report reflects the findings from a visit by Paul Roberts and Stuart Rye on 4th February 2010 and a follow-up review with David Johns on 24th February.”
So there was a subsequent follow-up review, do you recall that?
Alan D’Alvarez: I don’t – no, I don’t recall it.
Mr Blake: Thank you. If we look, staying with the right-hand side, there is an update that is provided at the bottom of that page, that isn’t present in the first version, and if we could zoom in on that update, it says:
“22nd February: Searches of the database since pilot launch and being run daily revealed one further incident of a dual settlement (this time without a banking transaction). There are over 100 branches live as of this date.”
So an update on 22 February, it’s happened again, this time without a banking transaction, so it’s a different type of transaction, suggests that the issues actually got worse by that stage.
Alan D’Alvarez: It shows that another incident was detected.
Mr Blake: Does it not show that the incidents – the matters got worse? I mean, it’s a new incident, a different type of transaction.
Alan D’Alvarez: Another issue – well, a similar symptom but a different trigger point.
Mr Blake: There are only 100 branches live at that stage. Is that what, that’s three out of 100 this issue is occurring in or known to be occurring in?
Alan D’Alvarez: So there’s three known incidents across two post offices out of 100 post offices that may have had, at that time, several hundred thousand transactions, yes.
Mr Blake: Well, in terms of branch numbers, what do we have? We have three out of 100?
Alan D’Alvarez: Well, two, because two was at Derby.
Mr Blake: Was that with two different counter clerks?
Alan D’Alvarez: Sorry, you was just saying in branches.
Mr Blake: Yes.
Alan D’Alvarez: Okay, so three incidents.
Mr Blake: Three incidents. Some to do – two of them or three of them to do with the banking transactions, one not to do with banking transactions. It seems to have got worse.
Alan D’Alvarez: They found another incident, yes.
Mr Blake: Is there a reason why you can’t agree that it seems to have got worse? Is there a technical reason for that?
Alan D’Alvarez: So if we picked up after two or three days two incidents after a week of going live of a pilot, two incidents, and then in another month you pick up another incident, yes, three is worse than two. It depends how you say has it got worse, ie is it just snowballing, is it just getting continually getting worse? I wouldn’t say so but it was another incident detected.
Mr Blake: Involving a different type of transaction?
Alan D’Alvarez: That’s what it says here, yes.
Mr Blake: An additional type of transaction?
Alan D’Alvarez: A transaction without a banking element.
Mr Blake: Can we go to the second page, please, on both of them. Is that possible? Am I asking too much? Excellent.
In fact, we can just look at the new one so 42175, if that can just be expanded and we move over to the second page. There is now no mention of that passage that general counsel had suggested should be removed. So that appeared in the first version and it now doesn’t appear in the background section. Do you remember that passage being removed?
Alan D’Alvarez: I don’t recall it, no.
Mr Blake: You received the email from general counsel saying, “Can this be taken out?”
Alan D’Alvarez: And that would have gone back to the authors. I would not – so I’m not the author of this document so I would not personally take it out. I would submit it back to Stuart Rye and Paul Roberts to deal with the feedback comments.
Mr Blake: Do you recall general counsel’s advice being taken?
Alan D’Alvarez: It appears to, if they’ve taken it out, yes.
Mr Blake: Can we look at page 5, please, of the right-hand side. We can take the left-hand side one off, I think. We’re just going to try and get it up on the left-hand side so we can compare the original.
We’ll just take a moment because it will help to see them side by side, I think. Just to clarify also, I’ve been asked to clarify, Mr Prenovost or Prevost, he was part of a team to lawyers, he wasn’t general counsel. I don’t know if that’s –
Alan D’Alvarez: We just call them Legal.
Mr Blake: Was he particularly senior in Legal, from your recollection?
Alan D’Alvarez: He’s the person that I was put in contact with so he was the interface I had for Legal. I don’t know his status within Legal.
Mr Blake: Thank you. There we have the two different conclusions. The original conclusion said:
“Overall, the actions taken to redress the Derby issue are appropriate. We believe the Counter Application fully supports the need to protect the integrity of financial transactions.”
Then the new version says:
“We believe the … Application with the identified fix fully supports the need to protect the integrity of financial transactions.”
Can you assist us with that change?
Alan D’Alvarez: Unless you applied a fix to the issue that we had, the integrity would not be maintained.
Mr Blake: Would a fix be applied to all counters or just those who raised the issue or just –
Alan D’Alvarez: All counters. It was part of the release and it also was mandated to be applied prior to going to any other post offices.
Mr Blake: Is it saying here now that the fact that we have applied the fix supports the need to protect the integrity of financial transactions –
Alan D’Alvarez: That’s what it’s saying, yes.
Mr Blake: – ie the fact that we found the problem and found a solution supports the need to protect the integrity of the financial transactions?
Alan D’Alvarez: Yes.
Mr Blake: So it’s quite specific, is it not, to this particular problem but we discussed earlier about whether this is signing off the system itself or the problem. Doesn’t it now seem to be even more specific to the problem rather than the system?
Alan D’Alvarez: If I – and I’m looking across at the conclusions and I just looked at 3.6:
“The business control at the Counter recognises that dual settlement risk is inherent by having two buttons that can initiate settlement. The business requirement is that the buttons operate on an exclusive basis – ie the use of one …”
So that’s not specific to this issue. That is demonstrating how, because you can have two ways of operating into the system, you cannot do them concurrently, because that could cause –
So the conclusions are wider, are showing protections that have been put as built in and designed in as part of the solution, which is not to do with that specific issue.
Mr Blake: So you still believe in this later report, later version, that it is signing off the integrity of the system itself?
Alan D’Alvarez: It’s validating our design and our implementation of that design that we should have confidence of the integrity of the system, yes.
Mr Blake: Can we go over the page, it’s page 7 on the right-hand side. It’s still part of the conclusions, 3.19 and 3.20. If it’s possible to do a similar thing on the left-hand side.
So you’ll recall I took you to 3.20, which is:
“Stock check is not a reliable method for catching issues due to the erratic nature of the stock check occurring.”
That doesn’t seem to exist in the later version, and also the recommendation that related to stock checking. If we could have a scroll down on the left-hand side if possible, to the recommendations. You’ll recall there was that recommendation:
“Consider advising the Post Office of the benefit of more effective stock control as an indicator of clerk errors or fraud.”
That seems to have been removed in the final version, as well. Are you able to assist us with why those were removed?
Alan D’Alvarez: Because I think, when you asked me that first question about stock check, I couldn’t understand, on that first one, why that’s relating to the integrity of the system. And I believe you highlighted that. You know, from our discussions, we couldn’t understand why this was part of this report, because it’s not to do with how system behaves; it’s how postmasters interacting with the system behaves. And it’s not to do with the subject matter in point.
So the advice from Legal was that if it’s not part of anything to do with integrity of the system, remove it.
Mr Blake: Do you think, looking back, it would have been helpful for Fujitsu to have taken that independent recommendation, which was to consider advising the Post Office of the benefit of more effective stock control as an indicator of clerk errors or fraud? Some other system, as an indicator of clerk errors or fraud, to back up whatever they were carrying out at that stage?
Alan D’Alvarez: I don’t even understand that advice. I don’t understand what that advice means.
Mr Blake: Thank you.
Sir, that might be an appropriate moment. If you’re content to take, say, a 15, 20-minute break now, we could go on after that and I’ll be 15 minutes.
Sir Wyn Williams: Yes, that’s fine, Mr Blake. Shall we say we’ll start again at 1.10, and then you will have completed approximately by 1.30. Is that what you’re telling me?
Mr Blake: I’ll definitely – I’ll conclude my own questioning by 1.30. There may be some questions. I may have covered everything. So, with goodwill, we will –
Sir Wyn Williams: All right. Anyway, with a limited time thereafter. Fine. All right.
Mr Blake: Thank you very much.
(12.53 pm)
(A short break)
(1.10 pm)
Mr Blake: Thank you very much. We’re now going to move to March 2010. Can we look at FUJ00094958, please. This is an email exchange of 26 March 2010. You are copied in, it’s forwarded to you by David Keeling. Who was David Keeling?
Alan D’Alvarez: David Keeling at that time was the customer services director.
Mr Blake: He says:
“Alan, Peter
“Please see Gavin’s note below and Dave Smith’s mail.”
Now, this is forwarding, at the bottom of that page, an email from David Smith. Do you remember who David Smith was?
Alan D’Alvarez: David Smith was Mark Burley’s – Mark Burley directly reported in to David Smith. I think he was the head of technology in Post Office.
Mr Blake: This email was in your bundle, have you read it recently?
Alan D’Alvarez: I was given it today.
Mr Blake: No, not this email.
Alan D’Alvarez: Oh, sorry, this one?
Mr Blake: This email you’ve definitely had for some time. We can go through it that’s fine. It starts off:
“Gavin
“I want to follow up our earlier telecon rather more formally.
“Whilst we don’t yet have a root cause of today’s issue given recent events it is difficult not to suspect that it might be related to the introduction of a change. Quite simply there have been too many incidents where poor execution of change has caused a problem in live. The situation demands that Fujitsu take action that is game changing whether that be increased rigour, an injection of [I think ‘different’] skills or a change in mindset.
“I also have to be concerned that we seem to be ahead of you and finding out for ourselves that there has been an incident in live rather than hearing from you. We have been here before and I will take a lot of convincing that this is not symptomatic of a reactive mindset. Again we need to see action that is game changing to a proactive style of management.
“The wider POL business and major stakeholders have been incredibly patient thus are. I believe we are now on the cusp of losing them and if we do then experience tells us that we could well end up on the front page of the Daily Mail. That will do damage to the reputation of both our businesses.”
This is quite an angry email, I think it’s fair to say. Do you remember it?
Alan D’Alvarez: I remember with David Keeling drafting a response on behalf of Gavin, yes.
Mr Blake: That’s the document that you were provided with.
Alan D’Alvarez: Yes.
Mr Blake: But do you remember the actual – seeing this email that was forwarded to you?
Alan D’Alvarez: Yes.
Mr Blake: Yes? Do you remember what “today’s issue” was?
Alan D’Alvarez: The Oracle. The Oracle bug that caused an intermittent failure to all the pilot post offices.
Mr Blake: So a significant issue?
Alan D’Alvarez: Yes.
Mr Blake: Do you think that a letter of this nature was justified?
Alan D’Alvarez: I think, as I – so I’d imagine that David Smith is under a lot of pressure from the business because having service outages at the pilot post offices for a period of time, which is losing business for those subpostmasters and Crown Offices, plus requiring compensation, David Smith will be getting a lot of pressure from his business and he wants answers from Gavin as to what are we doing about it.
Mr Blake: Before the break, we were discussing the independent report that was written and the conclusions of that report and it was your belief that that concluded, essentially, that the counter application had integrity or something along those lines. Do these kinds of incidents cause you to doubt such a conclusion, if that was their conclusion?
Alan D’Alvarez: This incident was preventing Post Office from trading. Nothing to do with the integrity of what was traded. The incident that triggered this email was post offices could no longer trade, so it’s nothing to do with integrity; it’s to do with the ability of the business, the business impact of not being able to trade.
Mr Blake: “Quite simply there have been too many incidents where poor execution of change has caused a problem in live.”
Isn’t that a broader complaint coming from the Post Office, that there had been too many incidents. Rather than just focusing on the one Oracle issue, this is a broader complaint, isn’t it?
Alan D’Alvarez: I believe that the Oracle issue triggered this. My understanding is that the previous issues that have been referred to, have been to do with maintenance releases that’s gone out to the Horizon estate, was my understanding at the time.
Mr Blake: So does nothing in this email cause you concern or to question a statement that the counter system had integrity?
Alan D’Alvarez: Had issues with integrity? No.
Mr Blake: We’ll look at the response. It’s FUJ00094958. This is a draft response. So in the email we’ve just seen, if we scroll up, or we may have passed it already but in the email we’ve just seen it says:
“Please see Gavin’s note below and Dave Smith’s mail. I’ve attached a draft note.”
So that is David Keeling has sent you this note in response. Do you recall this?
Alan D’Alvarez: He sent Gavin Bounds this note in response and I, with David Keeling, drafted this response.
Mr Blake: So you drafted the response?
Alan D’Alvarez: With David, yes.
Mr Blake: It starts:
“In response to your note of Friday, I do understand your concern regarding outages being contributed to by operational changes in the live estate and had already enforced the message of rigorous change control [access] across our technical and service management teams.”
I’ll move to the final paragraph, it says:
“The conflicting needs for deployment progression and stabilisation [that’s the issue we were talking about earlier] is always a challenge; which we recognise. However we are striving to demonstrate stability and continuing improvements to the existing Horizon Online branches.”
So that was a draft response. Are you aware of whether that response was sent or not?
Alan D’Alvarez: I assume it was, but I don’t know – if it was, I was not copied on it, or I don’t recall being copied on it.
Mr Blake: So nothing in the David Smith email caused you concern with regards to the integrity of the system. Can we look at FUJ00094958 – actually, sorry, FUJ00095107. Did you not think in that email that Mr Smith himself was questioning the integrity of the system?
Sorry, putting aside this that’s document on screen currently, the earlier communication from David Smith from the Post Office, did you not think that he himself was questioning the integrity of the system?
Alan D’Alvarez: I read that document that he was raising concerns that when changes are made in to the production system, there’s impact against the business, such as the Oracle issue which has caused a, you know, lack of trading. It didn’t – I didn’t read it as “I have concerns on the integrity of the system”. He said he is concerned that, as part of our process of releasing change into the production, we appear to disrupt the business.
Mr Blake: Because changes affect other parts of system, which then can cause errors in the system?
Alan D’Alvarez: I’m not going to come to that conclusion.
Mr Blake: We’ve heard in this Inquiry about things like code regression, when you change code it may have a negative impact somewhere else, or something along those lines. Was his complaint not that –
Alan D’Alvarez: Code regression?
Mr Blake: I’m taking that as an example. His email was expressing a frustration that the changes that were being made were causing other problems, wasn’t he?
Alan D’Alvarez: Impact to the business, yes, whether –
Mr Blake: Well, impact to the system, as well as the business.
Alan D’Alvarez: Could you put the email up again, please?
Mr Blake: Absolutely. If we go back to FUJ00094958. It’s the first paragraph of the bottom email on that page. He refers to the recent events. He says:
“Quite simply there have been too many incidents where poor execution of change has caused a problem in live.”
He’s not confining that to some sort of business problem. I mean, does it not read as though it’s concerned impact on the Horizon System and problems with the Horizon System, or –
Alan D’Alvarez: Okay, so that’s very wide. Okay, I don’t know what he meant by that.
Mr Blake: Let’s look at FUJ00095107. I’m only going to deal with this email briefly. Before it comes on screen, I can tell you it’s an email from Geoff Butts. You’re included in the copy list. It’s 6 April, so we’re now moving to April, and he says:
“I was asked to provide some criteria for restarting the High Volume Pilot.”
He sets out various matters there. You have added in the issues in square brackets, and by way of example, we can look at five. This is the Oracle issue, it says:
“A significant reduction in the number of Oracle 3136 errors seen in live.”
I think that’s your comment:
“What is the point of this? The main issue around Oracle 3136 errors was that it sent the BRDB into a non-recoverable ‘spin’ which led to catastrophic failures.”
Can you expand? Can you tell us what you were saying there?
Alan D’Alvarez: What I’m saying is you can’t have a reduction in the number of Oracle 3136 error messages. If that error message occurs, you have a catastrophic failure. Therefore, we have to fix it before we go forward. We can’t have a reduction.
Mr Blake: We have there nine individual numbered examples or issues. The number of issues raised here and your comment on, for example, that Oracle issue, again, it doesn’t sit well with a commitment to the integrity of the system at that time, does it?
Alan D’Alvarez: I don’t recognise the BAL failures. I don’t recognise what branch database failures mean in the context of (unclear). I clearly didn’t understand what number 3 was. Outages are not integrity related. Number 5 is not integrity related. Number 6 is not integrity related.
Mr Blake: So you were satisfied at this stage still –
Alan D’Alvarez: Number 7 is a wide – so, from my recollection, those that had a material impact or potential impact on integrity of the solution we took very seriously and we took to resolution.
Mr Blake: Were you open and honest and upfront with the Post Office about that?
Alan D’Alvarez: Yes.
Mr Blake: Always?
Alan D’Alvarez: Yes.
Mr Blake: Let’s look at POL00033100. This is the acceptance report for HNG-X. It’s dated in the bottom right-hand corner, 7 July 2010. We have, in this document, if we turn to page 9, we can see acceptance criteria. I only need to look at those very briefly. But by this stage, the stage of this report, 9 July, it says:
“there are no outstanding HNG-X High Severity Acceptance Incidents.”
That’s point (b) under 1.1. I’m going to look at page 31 in a little more detail. We have on page 31 medium Acceptance Incidents and there are two of those at that time. Thank you.
If we look at the table, it says, Low, 146, so there are 146 Acceptance Incidents there; medium, 2; high, 0. If we go over the page we can look at those medium ones at the bottom of that page. The first concerns patch management, and the second concerns cash declarations. Do you remember the cash declarations issue?
Alan D’Alvarez: I don’t recall it, no.
Mr Blake: If we turn to page 37, it there summarises that Acceptance Incident, the cash declarations issue. So TS058. Could I ask you to have a quick look at that and see if it brings back –
Alan D’Alvarez: Oh, okay, the cash holdings, overnight cash holdings. I recognise that one, yes.
Mr Blake: Can you tell us what that one was, in simple terms?
Alan D’Alvarez: So in simple terms – goodness. As I recall it in simple terms, that the process that – so the overnight cash handling process was a process that Post Office had to ensure that there was adequate cash levels in each of the branches or ATMs. They don’t want too much cash at a branch they don’t want too little cash in the branch because if they have too little they run out, they have to have emergency releases and things, that cost them money.
So part of the Horizon, and in the HNG-X system, will provide a statement called the ONCH, the overnight cash handling statement back to the centre to say this is the cash position at this branch and that will determine whether or not the Post Office deliver cash or pick up cash from that Post Office to make sure it has the right, adequate levels of cash for its business.
The issue here was, introducing HNG-X, the postmaster could do something out of the sequence to the process and the information going back to the centre about the cash position was incorrect. Now, the information of the cash position in the statement of accounts in the branch was correct and, therefore, it could create a situation where the cash position or the delivery or collection of cash to that Post Office was incorrect, and Post Office was very concerned that if a post office or an ATM ran out of cash it causes them extra costs because they have to go and do that.
So this was about a process, in as much as to – needed to be reworked so that the accurate cash position is put to the centre, so that they can do their cash replenishment.
Mr Blake: This was a medium severity incident and one that, as you said, the Post Office were not happy at all about. I’ll take you to the contemporaneous email on that it’s FUJ00097001.
Can we look at page 6 and 7 of that. You have been copied in to this email chain higher up. But if we can look at page 6 and 7. Phil Norton sets out, if we could scroll down and over to the next page, a whole load of reasons why he says it needs to be fixed, and I’ll just highlight for the sake of time the final paragraph. So it says:
“For all these reasons, this defect needs to be fixed as a matter of urgency and, contrary to the proposal by Fujitsu, without the raising of a CR. The Use Case ‘obtain daily cash on hand statements for SAP ADS’ was a retained functionality UC, and therefore HNG should operate in the same way as Horizon. It doesn’t and therefore this is a defect – not a Change. The paper from Gareth at the foot of this mail gives more detail of the issue and the proposed fix.”
This is still the same issue, isn’t it?
Alan D’Alvarez: This overnight cash handling – obtain daily cash on hand statements for SAPADS.
Mr Blake: I don’t want to rush you, you can have a little look above, if you like.
Alan D’Alvarez: Could you please.
Mr Blake: It says, “This [relates to] the feed to SAPADS having NO cash information …
“The operation implications of this are significant – both to the Network and the Cash Inventory Team.”
Or is this a different issue?
Alan D’Alvarez: I think that might be a different issue.
Mr Blake: So there’s another issue –
Alan D’Alvarez: It might be the same. We need to go back to the technical experts that resolved this issue.
Mr Blake: We can scroll on, if you like, to see if it assists you. Perhaps the final page, page 10, is the email from Gareth Jenkins. Do you recall the Post Office in June 2010, getting angry about what they called a defect, whether it’s the same issue as mentioned in that Acceptance Incident or not?
Alan D’Alvarez: It’s probably the same issue.
Mr Blake: Probably is the same issue. Do you remember the Post Office – this angry email from Phil Norton?
Alan D’Alvarez: And can you show me the email, please?
Mr Blake: Page 7. Page 6 and 7, bottom of page 6, top of page 7. You’ll see, if we scroll up to page 5, you’ve been copied in.
Alan D’Alvarez: This email here from Graham Allen?
Mr Blake: Yeah, you’re copied in there.
Alan D’Alvarez: So this email from Graham Allen you’re asking me about?
Mr Blake: Yes. What I’ve referred to as the angry email, it is simply the one I read to you before from Phil Norton which says:
“For all of these reasons, this defect needs to be fixed as a matter of urgency and, contrary to the proposal by Fujitsu, without the raising of a CR, the use [et cetera, et cetera]. It doesn’t and therefore this is a defect – not a Change.”
He’s specifically referring to this as defect.
Alan D’Alvarez: Okay.
Mr Blake: Do you remember the Post Office in this period, so June 2010, being concerned and raising those kinds of concerns?
Alan D’Alvarez: My – I don’t have a direct recollection, but Phil Norton is just debating, so I’m making an assumption that a response went back to Post Office saying this is a change, not a defect, and they come back saying they believe it is a defect and, on further investigation, it was determined to be a defect.
Mr Blake: Were they right to be angry about it?
Alan D’Alvarez: I don’t know if Phil was angry. I mean, can you show me the email again?
Mr Blake: Absolutely. It’s page 7. “Angry” is my word. It may not be angry but certainly insistent that it is a defect not a change, because of the significant implications it would have.
Alan D’Alvarez: So this email here, was I copied on this email?
Mr Blake: You were copied in the chain, not copied in the email.
Alan D’Alvarez: So he’s – I think he’s being forceful, that it needed to be fixed, as a matter of urgency, as a priority.
Mr Blake: Yes, especially as how close it is to ultimate acceptance of the system?
Alan D’Alvarez: Yes.
Mr Blake: Can we look at FUJ00097058, still in June 2010. Is this a document that you recall? It’s a report by somebody called Penny Thomas.
Alan D’Alvarez: I don’t recall it from the time, but I recall it from the pack that was presented to me.
Mr Blake: What was the issue with the ARQ return, in very simple terms, and what’s said here to be duplicate transaction records?
Alan D’Alvarez: So this is – I think I may have referred to this earlier, and mistook another potential issue, where we have an audit record that we keep, and there are times, just for – in any system, records can be sent twice and previous system that was dealing with audit records on Horizon had a facility to detect duplicate records and then to reconcile those.
My understanding of this is that the HNG-X did not do a similar thing and we had to kind of put in a facility to identify duplicate records.
Mr Blake: If we scroll down it says:
“A discussion took place between Alan Holmes, Gareth Jenkins and Penny Thomas.”
What were their respective positions?
Alan D’Alvarez: Well, Gareth Jenkins, again, SME. Penny Thomas, I am not sure but Alan Holmes was very much working in the requirement space and in also the – I think he was a business analyst.
Mr Blake: Point 3, one of the tasks/actions is:
“Gareth agreed to draft a statement for management review detailing the issue for onward transition to [Post Office].”
Then it says:
“A separate issue was also identified where a seemingly duplicated transaction had a different NUM and Gareth agreed to review of the detail.”
Do you remember that at all?
Alan D’Alvarez: No.
Mr Blake: No. If we turn over the page:
“We need to identify which cases provided with ARQ returns since the HNG-X application has been live have progressed to prosecution and identify whether duplicate records were included. We will need POL involvement to ensure all instances are covered. A very quick review identifies that both West Byfleet and Porters Avenue are included here, to what extent is not yet known.”
Are you aware of the relevance of prosecutions to this particular issue?
Alan D’Alvarez: No.
Mr Blake: Can we look at FUJ00097046, please. There is subsequent discussion of this issue, and let’s look at page 4 – page 3 scrolling down to page 4. So at the bottom of page 3, there’s an email from Penny Thomas. You’re not at this stage copied in. The subject is “Duplication of Transaction Records on ARQ Returns”. We have here number of ARQs affected, 112; 12 ARQs where court action is known, that’s the fourth entry – 12 ARQs where court action is known; number of cases, two; 8 ARQs where witness statement requested but has not been provided; number of cases, three.
Are you aware of the relevance of this ARQ information to prosecutions?
Alan D’Alvarez: No.
Mr Blake: Are you aware of concerns within Fujitsu copied in there, Gareth Jenkins from Penny Thomas about the relationship between this duplication of transaction records and its potential impact on prosecutions?
Alan D’Alvarez: I do not recall it.
Mr Blake: This is June 2010. Do you know how long this particular issue had been going on for?
Alan D’Alvarez: No. I’ve read this email trail for my pack, and whilst not an excuse, I always take the last two weeks of June off on holiday, so that’s my – I always do that. I was copied. Geoff responded, Geoff being my deputy. I do not recall this incident, and it may have been resolved because, in reading the email trails, there was a fix available. It may have been fixed by the time I returned and I wasn’t briefed about it. I just do not recall it.
Mr Blake: Okay, let’s look at page 3. The bottom email from Guy Wilkerson. Who was Guy Wilkerson?
Alan D’Alvarez: Guy Wilkerson was part of our – legal and commercial person that was assigned to the account.
Mr Blake: So he’s –
Alan D’Alvarez: Guy was more the commercial, and Jean-Pierre was the legal.
Mr Blake: Thank you. Guy says:
“Penny, would the additional transactions make any difference to the charges for a subpostmaster?”
If we scroll up, Penny Thomas says:
“These are original records which have been duplicated when copying to Audit Server. We are not suggesting that original records have been duplicated.
“If analysis was undertaken on the audit data some transactions would be duplicated; both plus and minus (we hope!). Analysis on stock units could be out as TI would show duplicated transfers and equally would TO. Cash on hand analysis would also be out.”
Are you able to assist us with what that means?
Alan D’Alvarez: That means specifically that if you don’t spot there’s a duplicate transaction in the audit record, not in the record of accounts in the Post Office – so that is – there’s no integrity question on the record of accounts. However, if you wanted to take a copy from the audit server of what happened, in that account, and you don’t spot there’s a duplicate transaction in there, then you may double count that and you may come to the wrong conclusion.
Mr Blake: Is that significant for the prosecution of subpostmasters?
Alan D’Alvarez: I don’t know.
Mr Blake: Is it potentially significant?
Alan D’Alvarez: I don’t know how this information was used.
Mr Blake: Is it significant to the reliability of the data?
Alan D’Alvarez: I don’t know how this information was used.
Mr Blake: If we scroll up to page 2, we have Gareth Jenkins responding. He says:
“Penny,
“No, that pretty well covers it. NB there is no guarantee that the duplicates are even complete sessions in which case the sum of all transactions may even be out.
“In summary, any detailed analysis of the finances of a Branch which is done with duplicate transactions without realising that there are duplicates (and so removing them) will give incorrect results.”
Does that strike you as a particularly serious problem, given that the Post Office was involved in the prosecution of subpostmasters and others?
Alan D’Alvarez: If this evidence – if this, sorry, information, this data, was used as part of that evidence, yes, it would.
Mr Blake: Scrolling up, from Guy Wilkerson:
“Gareth/Penny,
“I think we need Alan D’Alvarez or Geoff Butts to look at this – I’d hate to have POL raise this as an issue with an HNG-X Acceptance Board on Tuesday.”
Do you know why your name is being mentioned there.
Alan D’Alvarez: Because there’s an acceptance board for HNG-X and this is an HNG-X issue.
Mr Blake: Do you recall that that was, in fact, the acceptance board meeting where there was acceptance of the HNG-X system?
Alan D’Alvarez: I – it was a meeting that was scheduled for when I was on leave, but I don’t believe it took place until later.
Mr Blake: If we scroll up, were you back in early July?
Alan D’Alvarez: Yes.
Mr Blake: Penny Thomas says to Guy Wilkerson and Gareth Jenkins, copied in to you – would you have received emails? I mean, 2010, did you have probably a BlackBerry or something at that time?
Alan D’Alvarez: So I don’t know if I had a BlackBerry, certainly didn’t have an iPhone. So I would receive hundreds of emails a day. When I’m on leave, I would probably – I would have taken my laptop and I would have scanned just those emails that were directly sent to me, not cc’d me. Because if it’s sent directly to me, I may need to forward that email for someone to action on my behalf.
Mr Blake: So you made a specific decision to only review emails that were sent to you rather than copied to you?
Alan D’Alvarez: When I’m on leave, yes.
Mr Blake: How did you know they were copied to you until you’ve opened them?
Alan D’Alvarez: Because you’ve got “to”, and “cc”, and my email system is set up for those emails that are sent where I am on a “to” list, and the “cc” list goes into another folder.
Mr Blake: Even on your phone or whatever, you may have been taking on holiday?
Alan D’Alvarez: I didn’t have access to emails by my phone. As I said, I probably took my laptop on holiday.
Mr Blake: Your laptop. From Penny Thomas, she says:
“Guy, should I hold off advising my counterpart on this issue?”
So Penny Thomas was at Fujitsu. Presumably “counterpart” there is a reference to a counterpart at the Post Office?
Alan D’Alvarez: I don’t recall what Penny Thomas’s name is – sorry, role was.
Mr Blake: It says there “Security Analyst, Customer Services”.
Alan D’Alvarez: Okay.
Mr Blake: Can we scroll up. From Geoff Butts:
“Please do not make any communication on this issue with POL for the moment. We have been looking at this today and are waiting to determine if a proposed workaround is adequate. I will come back to you either tomorrow or Monday on progress.”
Then, above, Guy says:
“My nose never fails!”
We’ve seen in this email chain – yes, we’ve seen in this email chain reference to criminal prosecutions, a number of court cases, a number of cases where witness statements have been requested, et cetera. Given that the Post Office were prosecuting subpostmasters, assistants and others during this period, do you think it was right for Fujitsu to hold off telling the Post Office until they had a fix or a workaround?
Alan D’Alvarez: Typically what we would do was make sure we had the right information, because if you go to – if we go to Post Office and say “We have an issue”, you get a barrage of questions. It appears – only appears, I’ve got no facts from this email – that there’s a workaround. So to go to the Post Office and say “We have this issue, here’s a workaround”, is better than “We have an issue”, and then get 20 emails in the next two or three days of “What are you going to do? What can we do?” Et cetera.
I don’t know the scenario of what Geoff was managing at the time. It appears that he wanted to review the workaround so that we don’t go to Post Office and propose something that’s inappropriate, or it’s – and it’s robust.
Mr Blake: Was it a culture within Fujitsu at the time to do as you say: to hold off telling the Post Office until you found the workaround?
Alan D’Alvarez: No, until we have a clear definition. In Derby, we didn’t hold off telling the Post Office but we had a clear understanding there was an issue and what the issue was.
Mr Blake: There seems to be a pretty clear identification of what the issue is here, just not the workaround?
Alan D’Alvarez: It says, “To determine if a proposed workaround is adequate.”
So I think Geoff – I’m making an assumption, you had need to ask Geoff Butts this – it appears to me from this email he’s reviewing a workaround, and before he discusses it with Post Office, he doesn’t want to propose something that then we have to withdraw.
Mr Blake: Given that it might impact on whether information that’s being provided in a criminal court is correct or not, do you think it was right to hold off telling POL at that stage?
Alan D’Alvarez: If it was going to be within a day to go back with them, that could have been an appropriate action.
Mr Blake: Is there any indication there, it says “We’ve been looking at this today”, but is there any indication that it was going to happen then and there? The reason I ask is –
Alan D’Alvarez: You need to ask Geoff.
Mr Blake: – is it significant that there was the meeting to discuss the acceptance of HNG-X shortly after? Do you know if this issue was raised with the Post Office before acceptance?
Alan D’Alvarez: I would expect it would have done, with any either fix or workaround, yeah.
Mr Blake: You would have expected it to have. Did it?
Alan D’Alvarez: I don’t recall.
Mr Blake: Would it have been wrong if it hadn’t been raised with the Post Office before acceptance?
Alan D’Alvarez: Yes.
Mr Blake: Do you think Gareth Jenkins understood the significance in relation to the provision of that kind of information to a criminal court?
Alan D’Alvarez: I can only go by what he’s written there, but he appears to have understood that.
Mr Blake: From your discussions with him, and knowledge of him as a person, do you believe that he would have known?
Alan D’Alvarez: Technically he’s very able. I don’t know his knowledge of proceedings and evidence and that.
Mr Blake: And how about Penny Thomas?
Alan D’Alvarez: As I say, I don’t recall Penny. I can see, as you’ve pointed out what her role is, but she was in a different department within the account.
Mr Blake: What about those assurances about the integrity of the system that we’ve been discussing all day today? Do you think they were undermined by this issue?
Alan D’Alvarez: So the integrity of the cash account and integrity of the statement of record of what happened at a branch, this issue does not impact.
Mr Blake: But it impacts on information that’s purporting to show the integrity of the system, or a system that has integrity?
Alan D’Alvarez: It shows a management information system has duplicate records, yes.
Mr Blake: Quickly, can we go to FUJ00097080. This is the final email in that chain, 24 June, or the final email we have in that chain, 24 June. In fact that’s the same one as I’ve just been showing you.
Can we go back, please, to POL00033100. This is the acceptance report we’ve already looked at, 7 July. Shortly after this email chain, you’re presumably back in the country by this stage?
Alan D’Alvarez: Yes.
Mr Blake: Can we look at page 4. So version 1 was issued to the Acceptance Board on 23 June. That’s the bottom of that page. If we scroll down to 7 July:
“This has been updated to reflect changes presented to the board arising from new Acceptance Incidents”, et cetera.
I believe the meeting actually may have taken place on 29 June. Were you at the meeting?
Alan D’Alvarez: If it was on 29 June, I would not have been.
Mr Blake: Are you sure that you weren’t at this meeting? Would you have been invited to this meeting?
Alan D’Alvarez: I would have been invited, yes.
Mr Blake: Do you recall the final meeting where HNG-X was accepted? I may be wrong on the date of the 29th.
Alan D’Alvarez: I remember attending a meeting in Old Street for the acceptance, yes.
Mr Blake: At that meeting, before this was published and finalised, did you, or anybody at Fujitsu, to the best of your knowledge, raise that very issue that we’ve just been discussing?
Alan D’Alvarez: I don’t recall.
Mr Blake: Because Fujitsu were holding off telling POL until they may have a workaround. Do you recall anyone informing the Post Office either that there was a problem with the ARQ returns, or that that had been resolved?
Alan D’Alvarez: I don’t recall. It may have been fixed by that point.
Mr Blake: Do you think it would have been right to have told the Post Office, before HNG-X was accepted, about that issue?
Alan D’Alvarez: Yes, but if it was fixed it wouldn’t be part of the acceptance report.
Mr Blake: Well, it may be a concern, though. I mean we’ve seen that email from the Post Office about another defect, and I can’t remember whether – I used “angry”, you used “forceful” as the word. Wouldn’t raising this issue with the Post Office have received a forceful response of some sort that may call into question acceptance? Something of such significance?
Alan D’Alvarez: I don’t recall. As I say, Geoff obviously led this. I don’t know whether it was fixed immediately, how quickly it was fixed. The email trail which I read – which, when I saw this, seemed to suggest there was a fix. But –
Mr Blake: To use – I think I’ll borrow words from Mr Stein – that email chain we saw, it looks as though it’s a bit of a cover-up at that time. Would I be wrong to say that?
Alan D’Alvarez: I think you would.
Mr Blake: Not telling POL about a significant issue at this period, during this period, sounds quite significant. Am I wrong on that?
Alan D’Alvarez: I don’t know whether POL was told or not.
Mr Blake: But there’s an email chain that said “Hold off telling them”?
Alan D’Alvarez: For a day or two.
Mr Blake: Does it say “a day or two”?
Alan D’Alvarez: It says “until tomorrow or Monday.” That’s what I read, sorry. I may have misread that.
Mr Blake: It says, “I’ll come back to you either today or Monday on progress.” That’s Guy Wilkerson to Geoff.
Alan D’Alvarez: Or Geoff to Guy Wilkerson, I would have thought.
Mr Blake: “Guy/Penny, please don’t make any communication on this issue for the moment. We’ve been looking at this today and are waiting to determine if a proposed workaround is adequate. I’ll come back to you either tomorrow or Monday on progress.”
Alan D’Alvarez: Okay.
Mr Blake: So it’s not saying that they will come back to POL on that issue.
Alan D’Alvarez: No.
Mr Blake: “Tomorrow or Monday”.
Very finally, the very final document I’ll take you to –
Sir Wyn Williams: Before you do that, Mr Blake, is there any evidence that you have seen, written evidence, which shows whether or not POL was told?
Mr Blake: Are you asking Mr D’Alvarez?
Sir Wyn Williams: I’m asking both of you, in truth. You as Counsel to the Inquiry, in case there was a document one way or the other, but I rather doubt, given the nature of your questions.
So I’ll ask you, Mr D’Alvarez: have you ever seen a document which demonstrates that POL was told of this issue?
Alan D’Alvarez: I don’t recall seeing one.
Sir Wyn Williams: Right, fine. So if there is such a document, no doubt either POL or Fujitsu will disclose it.
Mr Blake: Yes. Ms Page actually has direct involvement in this matter to some extent, and she may be able to briefly address you on that.
Sir Wyn Williams: Well, I don’t need to be addressed. If there is a document, no doubt – which exists now – the relevant party will disclose it. And I’m just asking them to do it, basically.
Mr Blake: Thank you.
Final document is FUJ00097140. This is an email from Graham Welsh, the Migration Governance Manager. He says:
“For the avoidance of any doubt, please ensure this is cascaded to all the teams. All the hard work and long hours of the past months has achieved a position whereby we’ve entered into rollout.”
So 30 June 2010 is confirming that they had entered into rollout. So shortly after this email chain. Is that your recollection?
Alan D’Alvarez: What’s the date of this?
Mr Blake: 30 June 2010.
Alan D’Alvarez: I mean, I don’t recall the dates but I do recall we entered into –
Mr Blake: Are you aware of anything occurring between 24 June and 29 June in regards to that issue, in communication with the Post Office?
Alan D’Alvarez: I don’t recall anything, no.
Mr Blake: Thank you, sir. Those are all of my questions. I think Ms Page – no, she doesn’t.
Mr Whittam has a question or two.
Questioned by Mr Whittam
Mr Whittam: Very shortly, please, sir.
Sir Wyn Williams: Of course.
Mr Whittam: Richard Whittam on behalf of Fujitsu. I’d be grateful if we could have FUJ00094959 on the screen, please.
Just returning to this, the response you drafted to what Mr Blake described as the “angry email” from Dave Smith. When it comes up, if we look at the fifth paragraph down.
“It’s already been agreed and acknowledged that our prime objective is to minimise disruption and the ability of the branch network to trade. I am acutely aware of the support and tolerance of the postmasters and postmistresses in the HNG-X pilot, and recognise their patience. To that end, for the last four weeks we have been proactively calling the PMs in the pilot to gauge their experiences and to provide feedback on any issues they had logged to date. We have also attended the recent NFSP meeting in conjunction with POL programme.”
You drafted that. Was that accurate?
Alan D’Alvarez: I drafted that with Dave Keeling, yes.
Mr Whittam: Were you aware of the contact with postmasters?
Alan D’Alvarez: So contact with postmasters, I know there was a programme led by Graham Welsh, who was leading, with Geoff Butts, the Hypercare Team to get feedback. Myself and Mark Burley visited post offices and spent a morning with post offices that were having particular difficulties, to get their feedback and to give explanations as to what we’re doing, and moving forward. So I personally visited post offices with Mr Burley.
Mr Whittam: In relation to attending the NFSP meeting, in conjunction with POL programme, did POL have any involvement in that connection?
Alan D’Alvarez: When you say “the POL programme”, I believe that reference is to Mark Burley. So Mark had a standing – I don’t know what he called it, but he had a committee where he had representatives from I think both Crown Offices and postmasters. And we attended those meetings, as well, as – to give explanations on some of the issue that they are seeing what our approach is to resolution of those issues, so yes.
Mr Whittam: Thank you.
Thank you, sir. That’s all I ask.
Mr Blake: Sir, Mr Moloney has a very brief question as well.
Sir Wyn Williams: Yes.
Questioned by Mr Moloney
Mr Moloney: Thank you, sir.
Mr D’Alvarez, I represent subpostmaster Core Participants. I’ve just got one technical matter to ask you about, and that’s the swapping out of a base unit in a branch. You may remember that this morning Mr Blake asked you about issues in the Warwick branch in February 2010. At a meeting on 11 February it was noted that there were new BTS issues in Warwick and the base unit was to be swapped out, but that was noted as not setting a precedent, yes? Why were you concerned that it shouldn’t set a precedent?
Alan D’Alvarez: I wasn’t – when you say – it’s a case of if you have issues as an office, and because it was in the pilot site, we wanted to make sure, belt and braces, that once we’ve done that, we’d got a clean view as to what we’d done. I don’t recall the exact nature of why they chose to swap out the base unit. But if we state what we should do every time there’s an issue, we should make sure we swap out the base unit, that would be very disruptive and also costly, and it will serve no real purpose. But I do not recall the decision making to do that, other than it looks, in what I read this morning, like a – and as a belt and braces, let’s make sure they have a clean start so we eradicate every possibility of what could have caused the fault.
Mr Moloney: So, in essence, it would be important, in order to be able to establish what may have gone wrong?
Alan D’Alvarez: Yeah, and it was in the pilot. So in the pilot we put so much additional support, so much additional more analysis. We want to know everything that’s going on to assure ourselves that when we roll out to a large number of offices, that it’s as robust as we can make the solution and the system.
Mr Moloney: Would you, as a matter of principle, make a distinction between the pilot and, as it were, ordinary operations, post-rollout, in that regard?
Alan D’Alvarez: Yes. A pilot is a test phase.
Mr Moloney: But would, if there were problems experienced within an office, would substituting the process of swapping it out be a solution that one would try?
Alan D’Alvarez: It depends what the issue is.
Mr Moloney: So in some circumstances it might be right and in others it might not?
Alan D’Alvarez: Correct.
Mr Moloney: In some circumstances you might want to investigate whether or not there was an issue with the process before – well, after you’d swapped it out?
Alan D’Alvarez: Correct.
Mr Moloney: Thank you, Mr D’Alvarez.
Sir Wyn Williams: Is that it, Mr Blake?
Mr Blake: That is it, but I have an answer, and a document will be going up on CP View to Core Participants in due course. But I’ve certainly just seen correspondence from 30 June, so that is the day the email was sent about having completed rollout, or entered into rollout, from Fujitsu to the Post Office. So there was a – so we have the email from Graham Welsh that we saw of 30 June at 9.30 in the morning about having entered a position whereby they have entered rollout. There is an email that will be coming in due course – it’s unlikely to be an email for this witness, it will likely be an email for other witnesses – of the same day in which the Post Office are told by Fujitsu of the ARQ problem.
Sir Wyn Williams: Right. Fine. Well, no doubt that will be put in evidence when appropriate.
So thank you, Mr D’Alvarez, for giving evidence for the second time to the Inquiry. Thank you, too, for being flexible enough to give evidence today without having a proper lunch break, and I extend that thanks to everyone else in the room. That was primarily for my convenience, so I should own up to that. I have inconvenienced you all, but I’m grateful to you all.
So we’ll see you again at 10.00 tomorrow, yes, Mr Blake?
Mr Blake: Yes. Thank you very much.
Sir Wyn Williams: Thank you very much.
(2.08 pm)
(The hearing adjourned until 10.00 am the following day)