3 May 2023 – Anne Chambers
Hide video Show video
(10.00 am)
Mr Beer: Good morning sir, can you see and hear me?
Sir Wyn Williams: I can hear you but I can’t see you. Now I can.
Anne Chambers
ANNE OLIVIA CHAMBERS (continued).
Questioned by Mr Beer (continued)
Mr Beer: Thank you very much.
Good morning, Mrs Chambers. Yesterday, you gave evidence in the afternoon that ARQ data did not consist of all keystrokes made by a subpostmaster in a branch; do you recall?
Anne Chambers: Yes.
Mr Beer: You said that data was never captured.
Anne Chambers: Yes.
Mr Beer: That was page 160, lines 10 to 13 of the transcript. Before I ask you some questions about that, just by way of background for the Core Participants, in the closing submissions that the Post Office made in the GLO, the Group Litigation Order trial, the Post Office said this in their written submissions:
“Fujitsu does have access to a record of keystroke activity. The record must be downloaded from the counter rather than being contained in the events log. The counter log is called the POC log …”
In his Horizon Issues judgment, Mr Justice Fraser found that the audit data, as he described it, recorded all keystrokes performed in a branch by a subpostmaster. For those that are following that, paragraphs 906, 911(1), 911(6) and paragraph 995 of his judgment.
That conclusion was reflected in paragraph 15 of the Court of Appeal Criminal Division’s judgment in the Hamilton case, paragraph 15, stating:
“Fujitsu held data called ARQ data which contained a complete and accurate record of all keystrokes made by a subpostmaster or an assistant when using Horizon.”
I want to look at some of the documents that are referred to in support of what the Post Office said in the course of the trial, in those submissions.
Can we start, please, with POL00003233. Thank you very much. Now, if we just look at the top line of this, we can see this is a PEAK 273234. We can see under the progress narrative that it was raised on 21 August 2018 so that’s two years after you left and then, if you see the summary, it says the incident is a “Failed Drop & Go Top Up”, that doesn’t matter for present purposes.
Anne Chambers: Ct.
Mr Beer: If we go to page 3 of the PEAK, if we just, sorry, look at the bottom of page 2 first so we can see who is writing. It is Joe Harrison. Did Mr Harrison work with you when you were in the SSC?
Anne Chambers: Yes, he did, yes.
Mr Beer: Thank you. Was he a diagnostician too?
Anne Chambers: Yes, he was.
Mr Beer: He says:
“This is an instance of [and then he quotes a KEL]. Counter 2 did receive the [and then a message number] unsuccessful message but debited the customer [then he quotes some text] to the amount of [£30] anyway.
“As stated in the KEL ‘This may be an issue with script … or a user error. The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS.”
Then he says this:
“Here are the keystrokes and messages from the counter, which might help ATOS.”
Then if we scroll down and look at those, we can see that he has seemingly cut and pasted into the PEAK a series of text and can we look at the cut and paste that has occurred.
Anne Chambers: Yes.
Mr Beer: Can you see the first entry and then the majority of the remaining entries –
Anne Chambers: Yes.
Mr Beer: – refer to a button?
Anne Chambers: Yes, this when the postmaster touches the button on the screen and that moves it then – well, you can just see that the button has been pressed as it moves on to another screen.
Mr Beer: When you’re talking about the button being pressed, what are you referring to?
Anne Chambers: A virtual – have you had the chance to see a post office counter screen –
Mr Beer: Yes, so a tile on the touch pad?
Anne Chambers: A tile on the touch pad, yes.
Mr Beer: So where it refers to “button”, is that referring, is it, to tapping a tile on the touch pad?
Anne Chambers: Yes, or using the associated key on the keyboard, so, to that extent, yes, the button presses or the virtual button presses are recorded but not every single keystroke. So we can’t see here that a name has been typed in or that, you know, perhaps a name was typed in and then deleted or anything like that. So perhaps I misunderstood yesterday but I still say that not every keystroke is recorded. But for HNG-X, not for Legacy Horizon, we did explicitly ask for this extra level of diagnostics which helped us to see how the user was navigating the system at any point.
Mr Beer: So breaking that down, you remember the example you used yesterday of the £250 versus the £2,500?
Anne Chambers: Yes.
Mr Beer: If the postmaster – I think this was showing some cash in?
Anne Chambers: Yeah.
Mr Beer: If the postmaster wanted to show that they had received £250 in, and the system in the event showed two thousand £500 in, would you be able to tell, starting with Horizon Online that the postmaster pressed a 2, a 5 and an 0 and then Return or commit it to the stack, or whatever the button was to be pressed, rather than pressing a 2, a 5, and then 00?
Anne Chambers: You couldn’t see that level of detail. Obviously that information, whichever it was, would be captured and then stored on the system when the transaction was committed but then when it’s on the system, that would be the number that I am seeing. So I wouldn’t be able to tell that, at the point it’s actually being recorded by the system, it is not precisely what the postmaster had keyed.
Mr Beer: So if he said, “This system is showing that I was showing a receipt of cash of £2,500, I did not press a 2, a 5 and then 00, I only pressed a 2, a 5 and then a single 0”, you wouldn’t be able to tell from the keystroke data whether that was accurate or not?
Anne Chambers: No.
Mr Beer: All that would say is that the system shows that you pressed 2, 5 and 00 –
Anne Chambers: Yes.
Mr Beer: – because £2,500 is shown as cash coming in?
Anne Chambers: And that would also then be shown on the screen to the postmaster, so if he felt that the number was wrong, either because the system was now displaying it to him wrongly or because he’d miskeyed it, then you would expect that to be corrected at that point.
Mr Beer: Then breaking it down a little further there, you said that not all keystrokes were auditable.
Anne Chambers: Yes.
Mr Beer: What was the dividing line between those which were and those which were not –
Anne Chambers: Um –
Mr Beer: – ie what level of button was auditable? That’s a very imprecise question but I think you know what I mean.
Anne Chambers: Yes, I think any of the buttons that controlled the navigation around the system or where the postmaster – I mean, you can see the examples on here, where the postmaster was given a choice and had to choose “Yes” or “No”. When they were on the home screen and decided to go into a particular area of code, um, that’s –
Mr Beer: Trying to go into a particular area of code?
Anne Chambers: Sorry, yes, when they chose – sorry, that’s inaccurate. When they selected a particular function, for example, Postage or Bill Payment, other things would also be recorded early in the process. If they scanned a barcode, that barcode that had been read would be included in the logs.
Mr Beer: So you could see the order of events –
Anne Chambers: Yes.
Mr Beer: – is that right?
Anne Chambers: Yes.
Mr Beer: You could see the Pathway that a subpostmaster took?
Anne Chambers: For HNG-X, yes. And this was really useful for us, for diagnostic purposes, because we were able to see, you know – we’ll see that when we look at some of the specific examples, but we can see, yes, they started to do something and then they used a particular button to move out of it. Perhaps that’s not something that would normally happen but that doesn’t mean it’s wrong.
And so that was very helpful to us for diagnosing these problems because we could see the perhaps less expected paths that were being taken.
Mr Beer: You see four entries in, timed at 13.11.31, there is the word or the character string, “MSG10800: Check Parcels and Services Required”. Is that a record of a screen being displayed to the subpostmaster, essentially a pop-up that was displayed on the screen to the subpostmaster?
Anne Chambers: I think it was a question that he was asked at that point by a message on the screen. I can’t remember exactly how it was would have been displayed.
Mr Beer: What other ways of displaying it other than on the screen were there?
Anne Chambers: Sorry, that was the only way.
Mr Beer: So where we see the “MSGs” on here, the messages, is that a record – I’m calling them pop-ups, but essentially messages displayed on a screen to a subpostmaster?
Anne Chambers: Yes.
Mr Beer: Again, the same question: were there some such messages that were displayed and not auditable or were they all auditable in this way?
Anne Chambers: I can’t remember. I think they were all displayed but I’m not certain. They were not there for audit purposes, if you like; they were there as a diagnostic aid, as I said.
Mr Beer: What’s the difference?
Anne Chambers: Audit, I would feel is something that you would return to later and say, “This is precisely what happened and we have captured everything that has happened”. This data, although I think it probably does meet those criteria, it wasn’t designed with that in mind.
Mr Beer: How do you know it wasn’t designed with that in mind?
Anne Chambers: Because I and a colleague asked the development team when we had a meeting before, when HNG-X was being developed, and we said “Gosh, it would be really useful for us if we know what buttons were pressed and we know what messages were displayed”.
Mr Beer: Was any of this available to you for Legacy Horizon?
Anne Chambers: Not in the same form at all, no. You could get clues from the messages in the message store but it wasn’t designed – and there was a certain amount of audit – of diagnostic information, I think we discovered yesterday in the audit file, but this was very helpful at this level.
Mr Beer: This PEAK doesn’t refer to – Mr Harrison doesn’t refer to what he has cut in to the PEAK as a POC log. Is this in fact an extract from a POC log?
Anne Chambers: As far as I remember, yes. I don’t clearly remember all the filenames.
Mr Beer: What was a POC log?
Anne Chambers: A Post Office Counter log, a file that I think – I think there was one for each day, I think they were possibly kept for a limited time. It might only have been seven days, I can’t remember. They weren’t copied off the counter unless we needed to access them but they were there for diagnostic purposes.
Mr Beer: You said they were only kept, you think, for seven days. Do you mean kept on the counter for seven days but available in an archive after that time?
Anne Chambers: No, they weren’t archived.
Mr Beer: They were not archived?
Anne Chambers: No, I can’t now remember if it was seven days they were kept on the counter or if it was a longer period but it wasn’t a very long time and they were not taken off the counter and stored anywhere else, unless somebody in SSC went to get one for diagnostic purposes.
Mr Beer: Here Mr Harrison is cutting this in to the PEAK on 21 August, if we just scroll up.
Anne Chambers: Yeah.
Mr Beer: Yes.
Anne Chambers: Yes.
Mr Beer: He’s referring to events that happened on 31 July, so three weeks earlier?
Anne Chambers: So it was more likely then it was a month’s worth but I cannot clearly remember, I’m sorry.
Mr Beer: Was that by design, that they were only kept for a limited period?
Anne Chambers: I imagine so because, obviously, they could get fairly big and you didn’t want to fill up the counter file store more than you had to.
Mr Beer: Were they only available from the counter store and nowhere else?
Anne Chambers: Yes. They – yes, they weren’t kept anywhere else. Obviously, if somebody had gone – had looked at the same branch two weeks earlier and happened to have extracted that log, they might have it, but they would only be there – be anywhere else if somebody in SSC had specifically extracted them. That was the case certainly when I left in 2016. I can’t say what might have happened since then, of course.
Mr Beer: Can we look, secondly – that can come down, thank you – at POL00001835. Thank you.
This is a PEAK 209755. Earlier in time, you’ll see that it was opened by Mr Parker on 15 April 2011. The summary doesn’t tell us anything, six lines in from the top, it just gives the branch FAD code. If we look at the second entry, if we scroll down, we can see what the issue was:
“POL has a discrepancy with a postmaster regarding a transaction in Huddersfield for TPoS.”
What was TPoS?
Anne Chambers: I don’t know.
Mr Beer: “The branch thought that they [were] settling the transaction below to debit card but it has been ‘automatically settled to cash’. Branch thinks that something went wrong with their pin pad – debit card [transaction] declined but the branch didn’t notice.”
Then some details for the branch are set out:
“This was not noticed until the next day when they balanced and they then pulled off a transaction log and noticed the cash payment. A TfS call for this was logged on the day after the transaction … and NBSC and HSD both told the PM that it was user error.
“It has now been raised again via TPoS introduction managers – Fujitsu release managers, etc. To provide a sanity check please retrieve the counter log for node 7 on this date and see if we can add anything?”
In short, an issue had been identified with a subpostmaster trying to settle a transaction to a debit card but it had automatically settled to cash –
Anne Chambers: Yes.
Mr Beer: – and that was only noticed the following day –
Anne Chambers: Yes.
Mr Beer: – when the subpostmaster tried to balance and saw that the matter had been settled to cash?
Anne Chambers: Yes.
Mr Beer: Then if we can scroll down to Mr Allen’s entry, Dave Allen at the foot of the page there. Was he a colleague of yours –
Anne Chambers: Yes, he was a colleague.
Mr Beer: – doing the same work as you?
Anne Chambers: Yes.
Mr Beer: He says:
“Immediately after selecting ‘Sell Euros’ [message] ‘Transaction Prompt’ appears; this states ‘Transactions paid for using a debit or credit card will require mandatory ID’.
“I note this isn’t shown in the POC log for the Huddersfield incident.
“Subsequently, the Clerk selected Method of Payment – ‘Debit Card’, whereupon [a message] requests entry of the first 4 digits of the card’s PAN (the ‘Debit Card Prefix’).
“After entering the debit card prefix, [another message] ‘Clerk Instructions’ appears; this states ‘Do you wish to flag this transaction as suspicious for anti-money laundering purposes? If you select “Yes”, you must also complete [the] form [and a number is given]’ – the [postmaster] answered ‘No’ to this.
“After entering the Customer’s name and ID (passport) details, the Clerk is returned to the home screen which shows the ‘Total Due from Customer’ = £500.00 – as would be expected.
“At this point there is nothing to stop the Clerk settling to Fast Cash, even though ‘Debit Card’ has been selected earlier in the dialogue.”
Then this:
“The POC log confirms that ‘Fast Cash’ was indeed selected at this point.
“There is no evidence in the POC log of any PIN pad interaction at any time during this session and no evidence of any banking dialogue in the counter message log, and no evidence of the session being settled ‘automatically’ in some way, rather than by action of the Clerk.
“The counter logs can’t show us whether or not the Clerk actually took £500 from the Customer, in exchange for 540 Euros.
“Conclusion: the Clerk selected Debit Card as the method of payment early in the dialogue, but settled to Fast Cash at the end of the Session.”
Is this another example of being able to access the buttons pressed and the messages displayed that we saw in the previous PEAK, albeit Mr Allen has not cut in to his entry on the PEAK the text that supports what he has said?
Anne Chambers: Yes, he was using the same type of information from the POC log to give a narrative to what seems to have happened.
Mr Beer: So is it right, then, that the documents we have looked at show what selections, if I can use that word, a subpostmaster has made and what messages are displayed to the subpostmaster in the course of the session they are engaged in, as opposed to a record of every keystroke made?
Anne Chambers: Yes, and, yet again, I will say this is only for HNG-X.
Mr Beer: Can we look, please, at an example of where you have seemingly have had access to the POC log, FUJ00085913. You’ll see that this is a PEAK, dated 14 October 2015 – if we just scroll down for the first entry – with your name against it. It, in fact, concerns Bug 4 that we’re going to look at a little later, the Dalmellington bug?
Anne Chambers: Yes.
Mr Beer: If we see the summary, if we scroll up please, “Horizon – transaction discrepancies”. If we can skip, please, straight to page 5 of this PEAK, and if we look in the – sorry, page 3. Can you see, right at the foot of the page we’re looking at here, it says:
“keystrokes: Back Office, Remittances and Transfers, Delivery Scan your barcode”?
Anne Chambers: Yes.
Mr Beer: Can you help us, where is that information from?
Anne Chambers: I would have got that from the Post Office Counter log.
Mr Beer: That’s a similar sort of cut and paste by you from the POC log into this PEAK?
Anne Chambers: Yes.
Mr Beer: If we go forward, then, to page 5 –
Anne Chambers: No, actually, that’s not – hang on, that’s not me, because this is still an update that has been put on by –
Mr Beer: If we just go back to the foot of page 1.
Anne Chambers: Yes, this – yeah, this bit that’s highlighted at the moment is information that’s either been provided by it looks like it might have been provided by NBSC.
Mr Beer: Look at the foot of page 1. That’s where this entry begins, I think.
Anne Chambers: Yes. So this is information that has either been – that has been added by HSD, or whatever they were called at this point in time, based on information that they had received from NBSC.
Mr Beer: So did NBSC have access to the POC log then?
Anne Chambers: No. They must have asked the branch what they had pressed to get into this situation.
Mr Beer: Just go back to that entry we were looking at. If you look at the whole entry, that doesn’t look like it’s the record of a conversation in which a subpostmaster said what buttons they had pressed –
Anne Chambers: Um –
Mr Beer: – does it?
Anne Chambers: I would say, yes, it does. They have been asked by the Helpdesk, one of the helpdesks, specific questions and that is what they have answered.
Mr Beer: So where it says, “keystrokes” that’s a record of a subpostmaster saying it, is it?
Anne Chambers: Yes, because those are the – sorry, those are the buttons that he would press to do this process.
Mr Beer: Then if we go forwards, please, to page 5, and go to the bottom half of the page, please. We can see entries from you from 14 October onwards?
Anne Chambers: Yes.
Mr Beer: If we look at the third entry there, timed at 15.35.38, “Evidence Added”, and then is that a POC file reference code?
Anne Chambers: Yes, it is.
Mr Beer: What’s that saying that you have done?
Anne Chambers: I have, by this time, extracted the POC file for the day from the counter. I have examined it. I made some comments on it, which are further up the screen.
Mr Beer: Yes.
Anne Chambers: I have put it through the obfuscation process to make sure that no personal data is visible to unauthorised staff and then, once it was downloaded, it was automatically attached to the PEAK.
Mr Beer: Do we see the automatic attachment three entries, four entries on, where there’s an underlined entry reading “8th Oct poc.log”?
Anne Chambers: Yes, I’ve added two different logs, one for 8 October and one for 1 October.
Mr Beer: So if we had the PEAK system available to us now, that would be a hyperlink through to those files, would it?
Anne Chambers: Yes, I don’t know if those files would – underlying files would still exist or if they were deleted after a certain length of time.
Mr Beer: Look at it the other way, then, back in 2015, if you clicked on those, that would take you through?
Anne Chambers: Yes.
Mr Beer: So what was the purpose of putting the attachments in, in this way?
Anne Chambers: To make that available to fourth line support, who were GDC by this point.
Mr Beer: If we go to the foot of the page, please the second entry up from the bottom, you say:
“Routing to GDC [fourth line support, yes] to investigate by user was able to press and enter and settle the same ‘rem in’ basket multiple times. I have not managed to reproduce this.”
So can you tell from that entry, and in the absence of us having a POC log, the extent of the data that you were able to see.
Anne Chambers: I was able to see the button presses and, if we could just go up the page a little bit, I did put an update on to say there that I could see from the button presses that “Enter” had been pressed several times –
Mr Beer: If you keep going up, the second entry there at 17.42.11.
Anne Chambers: Yes.
Mr Beer: “I can see that the clerk pressed Enter 4 times …”
Anne Chambers: Yes.
Mr Beer: So thinking of the division that we made earlier or the evidence you gave about the division earlier, on what the POC log data did and did not record, this seems to suggest that delivery receipts were printed and then the clerk just pressed “Enter” four times?
Anne Chambers: That’s what the log showed, yes.
Mr Beer: So you could see a keystroke –
Anne Chambers: I could see those keystrokes, yes. I – yes. You could see the – and pressing that key would then cause the screen to move to a different screen, so it was – these were navigational keystrokes or keystrokes in response to messages, and so on, you could see.
When you asking yesterday, I thought you were asking about every key that was typed and certainly that was not all recorded.
Mr Beer: So if, in my example of committing cash to the account earlier of the £2,500 versus the £250, if the clerk, after they had typed £250, had hit “Enter” four times, would you be able to see that?
Anne Chambers: Um – it would – it would depend precisely how it was set up. You might be able to see “Enter” being pressed but I can’t be certain. I don’t know.
Mr Beer: What, if you can assist us, please, what on this occasion allowed you to see multiple button presses of the same nature?
Anne Chambers: I can’t –
Mr Beer: Is it the function being performed?
Anne Chambers: It’s the function being performed. I can’t remember what the question was that they were pressing enter in response to. I think it is recorded somewhere. It may well be – maybe it was something along the lines of “Has the receipts printed properly?” They pressed “Enter” for yes, which should then have taken them out of the process but, because there was an error situation, it went backwards and then printed a second delivery receipt and then they were asked again, has it printed? It had, so they pressed “Enter” for yes and, again, it was – this was an error situation but they were pressing cases “Enter”, which should have taken them out of the process but it wasn’t working as it should.
Mr Beer: Thank you, that can come down from there. That’s the only questions I ask about that topic from yesterday.
Can we go back to where we were from last night and explore your contact with subpostmasters. As we read yesterday in paragraph 212 of your statement, you said that the subpostmasters were not your clients. If you spoke to a subpostmaster, did you give them your name?
Anne Chambers: Um, I’d certainly give them my first name. Probably not usually my surname.
Mr Beer: Did you give them a means of contacting you?
Anne Chambers: No.
Mr Beer: Why was that?
Anne Chambers: Because they were not meant to have direct access through to third line support.
Mr Beer: How would they get access to you?
Anne Chambers: They could phone the helpdesk and ask that a message be passed to me and that did very occasionally happen.
Mr Beer: How very occasionally?
Anne Chambers: I don’t know. Three or four times ever, perhaps.
Mr Beer: In the 16 years?
Anne Chambers: Yes. It wasn’t something that – I mean, the whole point of having a support structure is that you’ve got the people nearer the bottom who are actually beavering away, resolving the problems and doing the investigations and I think almost any support system you have a certain amount of filtering with what direct contact there can be.
Mr Beer: Was there a duty or an obligation on you to speak to any subpostmasters or was it entirely at your discretion, if you thought it might help solve the problem?
Anne Chambers: It was at my discretion and I was slightly surprised there didn’t seem to be any guidance given on that.
Mr Beer: Surprised at who?
Anne Chambers: Perhaps at the general processes but, you know, I came into a team that was already up and running, working in their way and when you’re doing that, coming in as somebody new, you follow what everybody else is doing.
Mr Beer: We saw also yesterday that in paragraph 42(iv) of your statement you said that the MSU was responsible for liaising with the Post Office via BIMS reports, if there were errors which affected counter balancing?
Anne Chambers: If there were errors that affected the branch accounts or client accounts, bills being paid, information being fed through, they covered that area as well and also banking transaction discrepancies – not discrepancies, anomalies.
Mr Beer: As counter balancing was your specialist area, did that mean that you had more contact or a greater relationship with the people in MSU than others in the SSC?
Anne Chambers: Um, no, I think a lot of the counter calls – calls raised by MSU tended to be shared out amongst the teams, so I think a lot of different people would have had contact with them.
Mr Beer: Were the MSU involved in getting the Post Office’s approval for inserting or amending data into branch accounts?
Anne Chambers: We couldn’t amend data into branch accounts and, no, they weren’t.
Mr Beer: You said you couldn’t amend branch accounts?
Anne Chambers: Yes.
Mr Beer: What do you mean by that answer?
Anne Chambers: You couldn’t amend data that they had already written. All that we could do was to insert extra corrective transactions in the very few cases where that was seen to be the best thing to do to resolve a system problem that had already happen.
Mr Beer: Were MSU involved in getting approval for inserting extra corrective transactions?
Anne Chambers: No.
Mr Beer: Who was your point of liaison, therefore, back to the Post Office to get approval for such corrective amendments?
Anne Chambers: It went through whatever the particular change control process was at that point and, in practice, it would usually be the managers in the Service Management Team who would talk to people at Post Office.
Mr Beer: So who was your point of contact then, within Fujitsu first?
Anne Chambers: Well, I would – obviously, it changed over the years. The formal way of doing it was for me to fill in a form saying what was to be done, and so on, and then there were people who had to read that information and sign off that form. In practice, I would probably talk to my manager, a problem manager, one of the customer service managers. It just depended who had been involved with it. But there was a formal sign-off process, as well, which would always have included the SSC manager and one of the customer service managers.
Mr Beer: How did you find out whether the Post Office had approved the corrective amendments?
Anne Chambers: That would be added to the OCP, OCR, MSC – I can’t remember all the acronyms – but it was part of the formal process that there had to be a name and a sign-off on that. But I was not responsible for actually going and seeking that and making – I just filled in the form to start with and then other people were in charge of making sure that the correct sign-offs were done before I was then given the authorisation to do a change.
Mr Beer: You said yesterday afternoon, right at the end of the session of your evidence, that you knew of cases where the Post Office did not tell a subpostmaster that their financial data had been altered remotely by somebody within Fujitsu. That’s at page 207, lines 20 to 24. What was that knowledge based on?
Anne Chambers: Discussions, sometimes along the line of are Post Office going to – I wouldn’t necessarily be speaking directly to somebody within Post Office for this, although I know there’s one occasion when I did, at least. But there were several occasions where we’d say, “Will you notify the branch or shall we?” And they’d say, “No, we don’t think it’s necessary to notify the branch”.
Mr Beer: Why would they say or what reason did they give for it not being –
Anne Chambers: I don’t –
Mr Beer: – hold on – for saying it’s not necessary to notify the branch that their financial data had been altered remotely by somebody within Fujitsu?
Anne Chambers: That was their decision to make. I don’t know why they would make it. I would always have been happier if the branch had been fully informed.
Mr Beer: Why would you have been happier if the branch –
Anne Chambers: Because I always thought –
Mr Beer: – hold on. The transcriber has to write down what we say and it’s easier if I get the question out and then you answer.
Anne Chambers: Yes.
Mr Beer: I’m guilty of it as well, of interrupting you.
So did they give any reasons for not wishing to inform the branch that their financial data had been altered remotely?
Anne Chambers: I’ve seen it written down in one or two instances, I think, because they didn’t want to let the branch know that there had been a system problem.
Mr Beer: So deliberately keeping the existence of a Horizon system fault from the subpostmaster that it affected?
Anne Chambers: I think that certainly did happen on some occasions.
Mr Beer: Were you uncomfortable with this?
Anne Chambers: Yes, I was, really. I just felt it would be a lot clearer if everybody – if the branches knew when there had been a problem. I – if I spoke to a branch and there had been a system problem then I would say, “There has been a system problem”.
One particular instance I can remember where we – I know the branch wasn’t contacted was where, as far as we were aware, the branch was – didn’t know that the problem had happened, it had been brought to our attention because of an entry on the Reconciliation Report, and so undoing what had been wrongly recorded seemed like the best way forward and they may well not have been aware that they had had a problem in that case.
Mr Beer: When you refer to the “best way forward” do you mean the open and honest way forward?
Anne Chambers: The way to resolve it perhaps with fewest questions.
Mr Beer: Well, did it seem to you that, in this respect, the Post Office was applying an approach, so far as the subpostmasters were concerned, of the least said to them, the soonest mended?
Anne Chambers: I can’t speak for Post Office but I certainly got the feeling they did not want the – there were occasions when they didn’t particularly want the postmasters to know about problems.
Mr Beer: Can we look at some documents, please, starting with FUJ00142197. This is an email sent from you to Gareth Jenkins, and Andrew Keil and Mik Peach, on 10 December 2007.
Anne Chambers: Yes.
Mr Beer: If we read it together, you say:
“Gareth,
“We have a problem with a branch where a single SC line was written for 100 Euros (£484) with no settlement.
“This was in the middle of two RISP transactions and I suspect it’s another oddity in the LFS counter code.
“Initially it caused a harvester exception because some of the BlackBoxData info was missing, but that was corrected (so has gone to POLMIS?) and now the set of transactions for the day don’t net to zero, hence on the Incomplete Summaries report.
“I don’t know what to do about it. As it stands, when they balance I think they will have a gain at the branch. If we correct the POLFS feed so it nets to zero it will not be in line with the branch, and will probably cause problems in future.
“This might be a case for writing a corrective message at the counter but this has not been a popular approach in the past.”
Then you ask some questions.
Anne Chambers: Yes.
Mr Beer: You say that inserting a message was not a popular approach in the past. Is this a reference to what you were just describing or is this a different issue?
Anne Chambers: This is a reference to Post Office not wanting us to make corrections.
Mr Beer: So this is the same issue that we were just discussing?
Anne Chambers: Yes, this is.
Mr Beer: But this isn’t a communication between you and the Post Office, between Fujitsu and Post Office, this is an internal communication?
Anne Chambers: Yes.
Mr Beer: Did Post Office’s desire not to reveal to subpostmasters errors in the system have an effect on the extent to which you did insert corrective messages at the counter?
Anne Chambers: Um, I don’t think the alternative to writing the corrective message was doing absolutely nothing. Something had to be done about this particular problem because, as I said, in this case it was going to cause them potentially a gain, and they’d got the sort of equivalent of a – they would have the equivalent of a – now, would they? Yes, they would have had a receipts and payments mismatch or a non-zero line on their branch trading statement. Sorry, this – I’m trying to remember a long way back now.
Mr Beer: Yes.
Anne Chambers: Because they hadn’t balanced, there was still an opportunity where a corrective message at the counter to cancel out this incorrect line would have put them in the state that they should have been in, so it seemed worth considering that.
Mr Beer: What I’m asking is it seems that, by at least December 2007, the reluctance of the Post Office to reveal to subpostmasters, through the use of corrective action, errors in the system was having a chilling effect on you within Fujitsu about your willingness to do it?
Anne Chambers: Yes, I –
Mr Beer: Would that be fair?
Anne Chambers: Um, I mean, there’s the other position, which is that, you know, writing a corrective message, SSC making changes to counter accounts, you can understand why there was quite a reluctance to give us permission to do that as well.
Mr Beer: Why?
Anne Chambers: Possibly because, at some levels, it was thought that we didn’t have the ability to do that. I don’t know. I cannot speak for Post Office.
Mr Beer: Can we look, please, at FUJ00087194. This, I think, is related to the email that we just saw.
Anne Chambers: Yes.
Mr Beer: Just looking at the whole page first, can you describe what this document is?
Anne Chambers: Sorry, can I have a drink and a cough.
Mr Beer: Yes, of course.
Anne Chambers: This one of the change procedure documents, so an OCP which I filled in what has been proposed, why the change is justified, when it’ll be done, more details as to precisely what will happen, and then I’d already talked to Gary Blackburn at Post Office about it, so this is obviously after the discussion that I had with Gareth.
And then further down we can see that approval has been sought from Post Office through the formal route and there should also be sign-off by my manager.
Mr Beer: Thank you. So if we just read through it together:
“Write corrective bureau message for [then the branch code is given].
“A single … message … was written in error on 26th November … selling 1,000 US dollars, with no corresponding settlement line. To remove the effects of this message at both the branch and on POLFS, we will need to insert a new message to negate the effects of the original message.
“Justification: If the change is not made in the counter messagestore (before the stock unit is balanced on Wednesday), the branch will have an unexpected gain of £484 (or thereabouts …), and a receipts and payments mismatch. This gain would have to be resolved at the branch. There would also be an inconsistency between the branch and POLFS to be resolved. By correcting the problem locally, the branch may not be aware of the problem, and there will be no inconsistency between the branch and POLFS.”
You set out when it’s planned for. You set out some extra detail. Then you say:
“The message will include a comment to show it has been inserted to resolve this problem (this will not be visible to the branch).”
Skipping a paragraph, you say:
“Neither the new nor the old message will be included in data sent to POLFS.”
So I think this is a record to show that, despite the misgivings in the email exchange we looked at earlier, authorisation had been given. But you record twice on this document that, by doing it this way, the branch will not be aware of the problem and that the message will not be visible to the branch. Why was it important to record those two things?
Anne Chambers: Just so it was known that that was the case. It’s not saying that none of it would have been visible to the branch. They would have been able, if they’d printed their transaction log, they would have seen the first transaction and they would also see the equal but opposite transaction. They would see that but they would not have seen the comment –
Mr Beer: Who had done it?
Anne Chambers: – of who had done it.
Mr Beer: Why was it important to record that the “who had done it” will not be shown to the subpostmaster? Why were you writing that on here?
Anne Chambers: Um, just in case anybody at some point in the future wanted to know. I just tried to – you know, I wrote down as full a description as I could of what was happening and so, if there was a question at some point, we would know this particular fact.
Mr Beer: In writing it, were you giving some reassurance to POL “Don’t worry, this won’t be shown to the branch. They won’t see what’s going on here”?
Anne Chambers: I don’t recall that being my intention at the time. I certainly wasn’t doing anything to try to specifically hide it from the branch.
Mr Beer: Wasn’t that the effect of what you were doing, though?
Anne Chambers: I don’t think I could have added anything on that would – could I have made it obvious to them in some way? I’m not sure.
Mr Beer: Wouldn’t telling the branch assist them in future –
Anne Chambers: Yes.
Mr Beer: – in that if there had been a recurrence that was not picked up, then they might understand better how it had happened?
Anne Chambers: A recurrence would have been picked up by the same things that picked up this one. They hadn’t reported “This is a problem already”. If it had happened again, it would have been picked up by the same mechanism that picked it up this time.
Mr Beer: So are you saying that it’s best not to worry them with a fault in the system?
Anne Chambers: I wasn’t making the decision as to whether the branch should be informed or not. But, yes, by doing it in this way, maybe I was thinking, “Oh good, we can just get it sorted out before they balance, they don’t need to be bothered by it”. That probably – you know, if I had realised I was going to be questioned about it so long afterwards, I might have possibly made a different decision but that’s the decision I made back in 2007.
Mr Beer: Did the Post Office tell you to undertake this correction in a way that did not reveal this information to the branch?
Anne Chambers: I don’t recall them specifically saying that.
Mr Beer: Or did you do it in that way, as a matter of choice, because you knew that that’s what your client would want?
Anne Chambers: I cannot remember and I haven’t seen any documentation as to whether I had a conversation with Gary Blackburn as to whether he was going to contact the branch about this or not, and I don’t know what he said in reply. I think I probably would have asked him that question but I can’t remember.
Mr Beer: I mean, is what we see here – you undertaking the corrective transaction in a way that does not reveal the way in which the corrective transaction has been undertaken and who has done it to the postmaster – reflect the view that you received from the Post Office, that it was important not to reveal to subpostmasters any hint that there were issues with the reliability of Horizon?
Anne Chambers: I don’t think I took this action for that reason.
Mr Beer: Albeit the effect of your actions was not to reveal to a subpostmaster the person and the means by which the corrective action had been undertaken?
Anne Chambers: That was the result of what happened, given that Post Office chose not to talk to the postmaster.
Mr Beer: Can we look, please, at POL00023765. This is a PEAK from 7 December 2007; can you see that?
Anne Chambers: Yes.
Mr Beer: From the summary, the issue is with a branch and a branch FAD is given, “POLFS Incomplete Summaries Report”. You become involved in this later.
Anne Chambers: Yes.
Mr Beer: Can you recall or explain what an incomplete summaries report is?
Anne Chambers: Where the transactions, which had been for a day, for a branch, were harvested to be sent on to POLFS, which was their financial back end system. If the transactions didn’t net to zero then they would not be sent and we would have to investigate, you know, why there was an issue.
Mr Beer: If we go over the page, we can see, I think, you attaching some files, is that right, on 10 December?
Anne Chambers: Yes. This is the same branch as before.
Mr Beer: Yes.
Anne Chambers: Yes.
Mr Beer: We can see on the 11 December a couple of files or links to files, entitled “Details of how POLFS feed was corrected” and “Correction made to counter messagestore”?
Anne Chambers: Yes.
Mr Beer: Again, are they hyperlinks to documents –
Anne Chambers: Yes.
Mr Beer: – that we – I don’t think we have those. But anyway, if we go to the foot of the page, please, and look at Andy Keil’s entry. Was he a colleague of yours in SSC?
Anne Chambers: Yes, he was.
Mr Beer: He notes at 17.19.46:
“Worth noting that the branch did not have any issues with the mismatched transactions because this was fixed before they did the roll. The branch is not aware of this and it’s best that the branch is not advised.”
Anne Chambers: Yes.
Mr Beer: Again, is that a further reflection of a culture within the SSC of it’s best not telling the branches where such corrective measures are undertaken to their financial data by the SSC?
Anne Chambers: I think it’s just reflecting that, in this one specific case, Post Office had said that they did not want to – they were not going to contact the branch.
Mr Beer: You said “in this one specific case”.
Anne Chambers: Yes.
Mr Beer: You said earlier in your evidence and last night, that you were aware of cases where the Post Office did not tell a subpostmaster that their financial data had been altered remotely by somebody within Fujitsu. You’re not suggesting that this was the only example of it, are you? Rather, this is reflective of that wider practice, is it not?
Anne Chambers: This is the call that I had in mind when giving those answers. Very, very hard to remember now but I think, as time went by, we were aware that Post Office certainly did not always want to tell the branches of faults, and so on. But I wouldn’t say that this was fixed within SSC. As I’ve said before, if a branch had raised the problem themselves and we were talking to them and it – we knew it was a system error, then, yes, we would say so.
Mr Beer: What explains the difference of approach, then, if the –
Anne Chambers: Because the branch may not have been aware of this issue. It had only been – they hadn’t reported it as a problem. It had only been picked up on our internal reports.
Mr Beer: Did you feel uncomfortable with this?
Anne Chambers: Yes, I did. I would – I think I said earlier, I would rather that the branch had been involved in the discussions, so they knew what was happening.
Mr Beer: Is this another case of you just doing what was common practice and that which your client wished you to do?
Anne Chambers: I don’t think it’s that unreasonable to do what your client wishes you to do. As to whether it was common practice, this, you know, the whole process of making counter corrections was pretty unusual. It was not something that was happening every week, every month. They were very, very few and far between. So this was what our client wanted at the time. Perhaps it was me anticipating what our client might or might not want to do. But, personally, I would have been much happier if the branch was aware what was being done.
Mr Beer: Did the Post Office ever give any good or substantial or honourable reasons for not wishing for this material to be revealed to the subpostmaster?
Anne Chambers: I’m not sure that they gave us our reasoning – gave us their reasoning in that way, no.
Mr Beer: Was it a case, then, that they were – the reason was the least the subpostmaster knows about errors in the system, the better?
Anne Chambers: I think you have to ask what Post Office what their thoughts on that are. But I would say, yes, I did get that impression at times.
Mr Beer: How and from whom did you get that impression at times?
Anne Chambers: I think possibly once or twice I was on a conference call about a system problem with Post Office people, and I think I’ve seen at least one document where it’s minuted that they don’t want – they didn’t want to give opportunities for fraud, if postmasters became aware of certain issues.
Mr Beer: Can you just explain how revealing to a subpostmaster that a corrective action had been made to correct a bug in the system would give an opportunity for a subpostmaster to commit an offence of fraud?
Anne Chambers: I wasn’t talking about corrective actions there, I was talking more about overall discussion of system problems that had occurred. I don’t recall that ever being said. In fact, I’m sure that wasn’t ever said in any discussion as to a single corrective action at a branch.
Mr Beer: We’ve seen some evidence that people such as Penelope Thomas, Andrew Dunks, Brian Pinder produced ARQ branch data for the purposes of proceedings. Was there any method to alert them that corrective action had been taken to insert data or extra messages into a branch’s accounts?
Anne Chambers: If they had looked at all the PEAK calls for a branch, they might have seen those but I don’t know if that was part of their process. The OCR – the ARQ data would contain the – both the original transaction and the corrective transaction at the point at which they were done.
If the full unfiltered data was retrieved and inspected, then that would show the comment, for example, that I mention was added in this one. Certainly sometimes for counter corrections, and it wasn’t done consistently, but often we might use a counter number that didn’t exist to make it clear that it was something out of the ordinary, or a username, including SSC, again to show that it was something out of the ordinary.
That wasn’t done on this specific one and I cannot remember whether that was because I was specifically asked not to or I was just producing a transaction that was absolutely a mirror image of the one that shouldn’t have been there in the first place, and all I did was just change the signs on the values, effectively, and I left all the other data in there as it was. But I cannot properly remember my reasoning.
Mr Beer: What was the purpose of using a fictitious username?
Anne Chambers: To make – well, if it had “SSC” in it to make it clear that it was not done by somebody at the branch.
Mr Beer: Did you always use SSC or did you use other fictitious usernames that did not identify the SSC as having made the change?
Anne Chambers: It would always have been something that was very clear that it – I – as I say, I can’t remember without an example if it would have been something like SSC999, which would have been a valid username, or something else, but it wouldn’t have “Fred12” or something. It would have been something to draw attention to it, not to try to hide it.
Mr Beer: Yes, thank you.
Sir, that might be an appropriate moment for the morning break, as I move next to some examples of bugs, errors and defects.
Sir Wyn Williams: Yes, by all means. How long do you think is appropriate?
Mr Beer: Until 11.30, please, sir.
Sir Wyn Williams: Yes, fine.
(11.12 am)
(A short break)
(11.30 am)
Mr Beer: Good morning, sir, can you continue to see and hear me?
Sir Wyn Williams: Yes, I can. Thank you.
Mr Beer: Thank you very much.
I keep promising to get on to bugs, errors and defects but I’ve still got to cover something that I rather skipped over, Mrs Chambers.
Can we go back, please, to POL00023765. This was the PEAK that we just looked at about the corrective action.
Anne Chambers: Yes.
Mr Beer: If we can just look at the foot of page 2, please, we’ve got the message or the entry by Andrew Keil that we looked at in the morning session at 12 December, 17.19.46?
Anne Chambers: Yes.
Mr Beer: “Worth noting that the branch did not have any issues with the mismatched transactions because this figure before they did the roll [the rollover]. The branch is not aware of this and it’s best the branch is not advised.”
So is that recording that by 12 December, the fix had been applied?
Anne Chambers: Um, I assume so. Yes. I mean, it was in the OCP when it was due to be applied.
Mr Beer: Yes. If we just go over the page to an entry that I didn’t take you to, your entry on 14 December at 16.13.37. You say:
“As detailed above, the two POLFS incomplete summaries … have been resolved.
“The counter problem which caused the first issue has been correct by inserting a message into the messagestore, for equal but opposite values/quantities, as agreed with POL …”
Then you give the OCP reference.
Anne Chambers: Yes.
Mr Beer: “As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line with the branch. POLMIS received both the original message and the corrective message.”
But then you say this:
“Once the problem was corrected, there should have been no impact on the branch. However it has been noted that the stock unit of BDC had a loss of [£]1,000, which was generated after the correction was made. We have already notified Gary Blackburn at POL (email attached). This appears to have been a genuine loss at the branch not a consequence of the problem or correction.”
So by 12 December the corrective fix had been applied concerning a loss of $1,000. After that correction had been effected, a stock unit showed loss of $1,000. It was only generated after the correction was made and you’re saying that this appears to be a genuine loss of the branch and nothing to do with the correction.
Anne Chambers: Yes, I have obviously been thinking about this quite a lot. The loss was only generated when they balanced so that’s why it showed at that point, they hadn’t balanced before then.
I think my conclusion that it wasn’t a consequence of the problem may have been wrong. It wasn’t a consequence of the correction. I know that Mr Justice Fraser considered some of this and there was – I’m afraid we haven’t got the documents in front of us, but his view was that there had been two different corrections done and one of them was for the wrong amount, and I can – I disagree with that strongly, in that the correction that he thought was for the wrong amount didn’t affect the branch accounts at all. That was the correction to the POLFS back end feed.
But yes, the branch then had a loss in this stock unit. One possibility was that they had done a balance snapshot or something during the week and realised that, actually, they had got $1,000 more than they expected in that stock unit and had taken it out of there and put it into the main safe to see what happened. Another is that I’m now wondering if this line that was incorrectly written as an “SC”, serve customer, should actually have been another of these RISP lines, which was reversing a rem out, and so whether it now – can I get this right? Yes, that would – if it was the case, that would have had this effect.
But I agree now, certainly given those circumstances, it would have been far, far better to have talked to the branch at that point to try and work out whether they did have a genuine loss at the end of the day, whether it was something that they then could resolve themselves. I’m not aware that they ever phoned in about it. I don’t know if Gary Blackburn, who was aware of this, ever contacted them or checked to see if they did have any lasting problem, but no, this – it was not as a result of the correction, but it wasn’t the state that we wanted to end up in.
Mr Beer: On what basis did you, in the light of what you’ve just said, conclude that this was a genuine loss at the branch?
Anne Chambers: Um, I don’t know. I mean, because I had checked very carefully and I could see that my correction had done precisely what I intended it to, which was to remove this rogue SC line, which should not have been written. It’s only now I’m wondering if, when it was written, it should actually have been another RISP line but I can’t prove that at this point.
Obviously, if the branch had raised another – a call saying that they’d got an unexpected loss, “What on earth has been going on”, then that would have been investigated and followed through but, to the best of my knowledge, they didn’t.
Mr Beer: Did it occur to you at the time that the amount of the correction, the value of the correction that you had made, was equal to the value of the loss that was now being shown?
Anne Chambers: Yes, of course it did. Which is why I checked and double checked and triple checked.
Mr Beer: And therefore might be a relationship between the two?
Anne Chambers: Yes.
Mr Beer: Isn’t what you’ve just said though, to put the burden back on the branch, to say they need to complain again, they’ve got to go through the whole rigmarole of going to NBSC again?
Anne Chambers: They hadn’t actually complained at all.
Mr Beer: No.
Anne Chambers: They didn’t raise the original call.
Mr Beer: Okay, they would have to go through a rigmarole for the first time, then?
Anne Chambers: Yes.
Mr Beer: Okay, we’ll move on. I’m going to ask you about as many of Bugs 1, 2, 3, 4, 5, 6, 7, 8, 10, 19 and 23 as identified in the appendix to the judgment of Mr Justice Fraser in the Horizon Issues trial as time allows today, and then I’ll revert to the process when we meet again for the Phase 4 evidence.
I’m not going to rely on his findings for the purposes of asking you questions, not least because we have more material than was apparently made available to Mr Justice Fraser. Just so you understand, what I’m going to try to do is firstly seek to understand in general terms what the nature of the relevant bug was, in a very high-level summary, then identify the issues that I would like to try and explore with you, then run through the material in chronological order that concerns that bug, and then explore any issues that are left that haven’t been addressed.
I’m not going to explore the bugs in chronological order, simply do them 1, 2 and following.
Anne Chambers: Yeah.
Mr Beer: Before we get to that, would you agree that not all errors in the Horizon System were caught by the automated processes set up by Fujitsu to detect errors?
Anne Chambers: Yes.
Mr Beer: You tell us in your statement – no need to go there, it’s paragraph 41 – that:
“From around 2007 a real-time monitoring system was developed by the SSC to alert us to system-wide problems, for example a large number of debit card transactions failing. This system was tweaked and expanded over the years.”
What was the name of that system?
Anne Chambers: The SSC monitor? Um, I can’t properly remember.
Mr Beer: Who monitored it?
Anne Chambers: We took it in turns. We had an SSC monitor monitor.
Mr Beer: Was that one person a day or a shift?
Anne Chambers: Yeah, a day.
Mr Beer: Who developed it?
Anne Chambers: I think John Simpkins probably did most of it but then, if other people had good ideas of how things could be monitored, they got sort of added into it. It was more monitoring of the back-end systems, not the counters themselves although, obviously, banking transactions, and so on, were going all the way through the system.
Mr Beer: When you say monitoring of the back-end systems are you referring to POLFS there?
Anne Chambers: No, I’m –
Mr Beer: You’re referring to Fujitsu back-end systems?
Anne Chambers: Fujitsu back-end systems.
Mr Beer: So how did it operate? What was its coverage?
Anne Chambers: I cannot remember any – much detail at all. It was just wherever there was a useful source of information, perhaps, for example, about the number of debit card transactions going through, they would have a response code on them to say, if it had been a successful payment or otherwise, we could monitor how many were going through a particular point in the system with some sort of failure/error code. And if it exceeded a certain threshold, then it would go red instead of green and that would encourage somebody to see what was going on.
Mr Beer: So it was essentially a sort of pattern analysis?
Anne Chambers: Yeah, for that particular instance.
Mr Beer: Can you help us, other than failed debit card transactions, what, if anything, else it picked up?
Anne Chambers: Banking transactions, which were actually a separate system. I can’t now remember the details, I’m sorry.
Mr Beer: There wasn’t such a system before 2007; is that right?
Anne Chambers: Not sort of trying to pick up problems before anybody had reported to them – them to us in some other way, yeah.
Mr Beer: But, in any event, the system didn’t itself proactively identify all bugs, errors and defects?
Anne Chambers: Not at all, no.
Mr Beer: Was Fujitsu essentially reliant on, therefore, a problem occurring within the live estate causing a discrepancy or a loss, and the subpostmaster raising it through the NBSC or the Horizon Support Desk?
Anne Chambers: We had all the reconciliation reports that ran overnight, so that was the main way of finding financial inconsistencies on the system.
Mr Beer: So there was the reconciliation reporting system?
Anne Chambers: Yes.
Mr Beer: Did it nonetheless remain the case that the majority of bugs were picked up through subpostmaster initiated action?
Anne Chambers: Um, I mean, obviously there were problems to be investigated throughout the whole system, all the back-end stuff as well, but if we’re talking specifically about counter balancing problems, which were only a very small proportion of the overall calls that we were handling, um, then I would say it was probably about 50:50 inconsistencies being reported by – on the reconciliation reports or branches reporting that they had a problem in a particular area.
Mr Beer: You said that counter balancing was only a small proportion?
Anne Chambers: Oh, yes.
Mr Beer: But it was a very significant issue for the subpostmaster concerned –
Anne Chambers: Yes, of course.
Mr Beer: – potentially?
Anne Chambers: Yes.
Mr Beer: Did you realise that at the time, did you acknowledge that at the time, that the consequences for a subpostmaster may be very extreme indeed?
Anne Chambers: I don’t think we – certainly, as I think I said yesterday, I didn’t realise initially that – how – really how the Post Office subpostmaster structure worked and that they were financially responsible. Obviously, some of problems would have been at the bigger Crown branches, which Post Office were responsible for. And there was always this huge difficulty in distinguishing where a problem is caused by something in the system and the – certainly more than just a possibility that it is caused by some inaccuracy of processing at the branch itself, the user input.
Mr Beer: Did you and others in the SSC treat counter balancing issues any differently because of a recognition that the consequences for a subpostmaster may be very direct and personal?
Anne Chambers: I don’t think that would mean that we would necessarily give it – you know, sort of put it to the very top of the heap. You could argue that it’s actually extremely important that a branch or a whole series of branches can trade. If they’re not able to trade, that is also – has serious consequences for all of them.
If the entire estate can’t do banking transactions that obviously also has a severe impact on the whole estate and so, to some extent, I think those type of problems may have been seen as more important – not more important but would possibly require faster action than a discrepancy call from a single branch.
I mean, I do see now that, yes – I am well aware of the impact that these problems have had. But it was so hard to distinguish between business issues and potential system issues, and we would look for every possible sign of a system issue. But if there wasn’t one, without knowing what had actually taken place at the branch, you can’t do more.
Mr Beer: Would your view have been different as to the relative importance accorded to bugs, errors and defects that may have affected the ability of the system to continue to trade, ie financial issues, on the one hand, and issues that may affect the continued employment or suspension, civil proceedings against, criminal investigations and criminal proceedings against, subpostmasters, on the other, if you had known more about how the Post Office had treated the subpostmaster contract as meaning that subpostmasters were responsible for all losses?
Anne Chambers: Yes. I feel we should perhaps have been warned if the result of us looking at a single call over a single day, or whatever, was going to – could result in action being taken against a postmaster with, I don’t know – I don’t know how much extra investigation was ever done.
Mr Beer: In the early days, say between 2000 and 2006, did you not realise, therefore, that the conclusions that you reached, the nature of the investigations that you undertook that preceded them and which you wrote up on a PEAK, could result in the next day a subpostmaster being suspended and locked out of their branch?
Anne Chambers: No, I don’t think we did realise that. I assumed there would be a huge amount more investigation and double checking of the figures and everything else.
Mr Beer: Double checking by whom?
Anne Chambers: I assume people in Post Office would be doing that.
Mr Beer: Can we turn to Bug 1, please, the payments and receipts mismatch bug. Can we start with my sort of summary of it. Would you agree with the following summary of the bug: firstly, it was a Horizon Online bug that occurred in 2010?
Anne Chambers: Yes.
Mr Beer: Secondly, it occurred when a subpostmaster tried to roll over a stock unit with a discrepancy?
Anne Chambers: Yes.
Mr Beer: Thirdly, the system would ask the subpostmaster if they wanted to transfer the discrepancy to the local suspense account?
Anne Chambers: Yes.
Mr Beer: If the subpostmaster cancelled the rollover, the discrepancy was zeroed from the location cache but nothing was written to the branch database?
Anne Chambers: Yes, I believe that’s true.
Mr Beer: If the subpostmaster then tried to roll over, the stock unit would be rolled with the corrupt local cache missing the discrepancy?
Anne Chambers: Yes.
Mr Beer: That would therefore create a receipts and payments mismatch?
Anne Chambers: Yes. Although I think that receipts and payments mismatch wasn’t actually picked up until the end of the following period.
Mr Beer: The issues that I would like to explore with you, please, are, firstly, why it appears that only significant action was taken in relation to this bug from September 2010 onwards when, firstly, the PEAKs in relation to it had been raised in February 2010 and, secondly, Mr Jenkins appears to have been aware of the bug in May 2010 when he noticed a Windows NT event; and then, secondly, what was done to ensure that all branches that may have been affected by the bug had been properly identified.
Anne Chambers: Right. I’ve got several things to say in response to that. Firstly, from everything I’ve seen about this bug, I was not involved in the investigations in September. So, really, everything I’m going to say is based on what I have read since. I have no memory of it.
I haven’t seen any evidence that suggests it was – that it did occur before September. I know there were couple of receipts and payments events which Gareth flagged, and there’s an email about that earlier in the year, and there was also at least one other problem that occurred during the HNG-X pilot, which was roughly the first six months of 2010. But they were different underlying causes and I’m not aware that this specific problem, which resulted in a receipts and payments mismatch had been seen or reported before September.
Mr Beer: That’s a very helpful general answer. Can we look at material then, the chronology of events. There are about ten steps in the process that I would like to ask you about, but there are about another 20 steps in the process but I’m going to ask other witnesses about those or suggest that they’re adequately established through the documents themselves.
Can we start, please, with FUJ00081064. Can you see that this is PEAK 0194381.
Anne Chambers: Yes.
Mr Beer: It was opened on 10 February 2010?
Anne Chambers: Yes.
Mr Beer: You can see from the summary “Counter APP”; what does “APP” mean?
Anne Chambers: Application?
Mr Beer: Total receipts £250,016.45, total payments £200,016.45. Then if we see from the first entry that summary is included, so a £50,000 discrepancy; do you see that?
Anne Chambers: Yes.
Mr Beer: So this is showing a mismatch, is this right –
Anne Chambers: Yes.
Mr Beer: – between receipts and payment –
Anne Chambers: Yes.
Mr Beer: – of £50,000?
Anne Chambers: It’s reporting a mismatch, yes.
Mr Beer: Now, I don’t think you, as you have said, ever became involved in this PEAK, so far as I can see; is that right?
Anne Chambers: I can’t remember unless I go down the –
Mr Beer: Yeah, if the operator could just scroll through, please, you’ll see I think your name doesn’t appear on it.
Anne Chambers: Okay. So it’s been sent off to GDC, who are providing fourth line support.
Mr Beer: Yes, and if we scroll down, please, I think we can see that your name is not on it.
Anne Chambers: Yes, okay, I do now remember this. I mean –
Mr Beer: This document?
Anne Chambers: I remember seeing this document before. Yes.
Mr Beer: We can see, if we go back up to the top of the first page, that this becomes “KEL ballantj1759Q”?
Anne Chambers: Yes.
Mr Beer: We can see that under the “All references” section, yes?
Anne Chambers: Yes.
Mr Beer: Can we look then, at “KEL ballantj1759Q”, that is POL00029425. This was created, we can see, by your colleague John Ballantyne on 12 February 2010 –
Anne Chambers: Yes.
Mr Beer: – and last updated by you on 17 May 2011?
Anne Chambers: Yes.
Mr Beer: The way a KEL is written, you can’t actually tell what Mr Ballantyne originally wrote and what you changed subsequently; is that right?
Anne Chambers: You can’t see on here. The old ones were kept but I’ve no idea if they still exist.
Mr Beer: So the text on here, we can’t see what was his work and what’s your work?
Anne Chambers: No, no.
Mr Beer: I don’t suppose you now recall what changes you made?
Anne Chambers: I may recognise some of my –
Mr Beer: Your style?
Anne Chambers: – style, but I’m not sure.
Mr Beer: You’ll see it cross-refers, in about the tenth line there, back to the PEAK we just looked at, yes?
Anne Chambers: Yes.
Mr Beer: If we scroll down, please, it states the problem:
“This event is generated when the payments and receipts totals do not match on one of the counter balancing reports. This indicates a software error or data corruption.”
Anne Chambers: Yes.
Mr Beer: So it continues:
“[This] has been caused in the past by …”
Then three possibilities are set out, yes?
Anne Chambers: Yes, yes.
Sir Wyn Williams: I’m sorry to interrupt you, Mr Beer, but I’ve had a message to say that I’m no longer on the screen. I’d just like to assure anybody who is looking that I’m still here and the problem with me being on the screen is being seen to.
Mr Beer: Yes, thank you, sir. We’re going to be looking at lots of documents at the moment so you wouldn’t have been seen, in any event, because when we look at a document, you disappear.
Sir Wyn Williams: That’s all right, then. That’s fine.
Mr Beer: The solution is set out:
“SMC/counter eventing team: raise a B priority call and send to SSC if you see this event, unless it is from a training counter …
“SSC: Instances of this error must be investigated. If the error is as a result of a new problem, please add the details to the list of causes above.
“The branch accounts may need to be corrected. See [another KEL] for advice on how this has been done for a previous problem.”
What do you understand “The branch accounts may need to be corrected” to mean?
Anne Chambers: I don’t know now and, when I covered this in my witness statement, I hadn’t seen the “wrightm” KEL. I have now and it doesn’t cast any light on it so I’m sorry but I don’t know why that’s there.
Mr Beer: You made a point in your witness statement I would need to see “wrightm”. We’re going to look at the “wrightm” KEL in a moment.
Anne Chambers: Yeah.
Mr Beer: So you don’t understand what that means?
Anne Chambers: Unless it’s to – referring to the corrective action that may or may not have been taken for the September bug, where they pressed cancel at a certain point.
Mr Beer: Who was this direction to correct the branch account addressed to?
Anne Chambers: It’s saying it to SSC, I believe, but don’t think I – I’m just about certain I did not put that in there. So I’m not entirely clear why it is there.
Mr Beer: By what method would you identify which branch accounts need to be corrected?
Anne Chambers: Once you had a full understanding of the specific problem and its consequences.
Mr Beer: By what method would they correct the branch accounts?
Anne Chambers: It would depend to the problem and its consequences.
Mr Beer: In your witness statement – no need to turn it up – paragraph 66, you say:
“Post Office would have been informed of each instance. I am not sure whether this was via a BIM or some other route. Fujitsu would not have contacted branches directly unless the branch had raised the call in the first place.”
By that, are you saying that the Post Office would have been made aware of each of the individual cases where this issue affected a subpostmaster or are you saying that the Post Office would be informed that there was a systemic problem?
Anne Chambers: Um, as I said, when I wrote this section I was working a little bit blind, given that I had no direct involvement with this. We wouldn’t have told Post Office about the office snapshots problem. Actually, they probably were told about it because everything in the pilot was closely monitored, but the office snapshot one there, that was false reporting of a receipts and payments mismatch because it didn’t take the transfer into account.
Obviously, the stuff that had to be done for the September problem was a major problem, which was all followed through at the time.
Mr Beer: Followed through by?
Anne Chambers: I wasn’t involved but I believe you’ve got some more documents about it.
Mr Beer: So in that passage in your witness statement, in which you said, “Post Office would not have been informed of each instance” – sorry, “would have been involved (sic) of each instance”, you’re not sure whether this was via a BIM report or some other route, “Fujitsu wouldn’t have contacted the branches directly unless the branch had raised the call in the first place.”
Is that essentially a reflection of the division of approach that you described to us yesterday, ie what determined whether or not you contacted a branch or not?
Anne Chambers: Yes. I believe so.
Mr Beer: Ie it depended on whether the branch had initiated the issue?
Anne Chambers: Yes.
Mr Beer: Can we turn then to the wrightm…J KEL that you said in your witness statement you needed to look at. That’s FUJ00081608.
This is the wrightm33145J KEL –
Anne Chambers: Yeah.
Mr Beer: – that we saw referred to in the KEL that you had last updated on 17 May 2011.
You’ll see that this KEL is not raised until 23 September 2010 –
Anne Chambers: Yes.
Mr Beer: – which is seven and a half months – I hope I’ve got the maths right on this occasion – since the PEAK that we were looking originally at 10 –
Anne Chambers: Yes, but that original PEAK was the office snapshot problem, not the same problem that happened in September.
Mr Beer: Why do you restrict the previous PEAK to only the office snapshot problem?
Anne Chambers: I don’t. It was originally raised for the office snapshot problem but then when there were other issues that could cause receipts and payments mismatches, it was useful to include them on there so that somebody subsequently checking that same error message could see what had happened in the past and what was – and it did say if any new problems come in with this is symptom, it will need to be investigated again.
Mr Beer: So the sentence that we saw in the ballantj KEL can’t have included originally the cross-reference to there is KEL –
Anne Chambers: No –
Mr Beer: – because this KEL didn’t exist at that time –
Anne Chambers: No, of course not.
Mr Beer: – when it was written?
Okay, we’ll come back and look at this KEL in detail at a moment. If we just go back to the chronology, then, because this isn’t raised until September. I just want to see what had happened in the interim. Can we look at FUJ00081062, please. This is an email chain, I think, all dated 6 May, certainly the part that I wanted to refer to. If we look at the bottom of the page, please. Thank you. If we scroll up so we can see who it’s from and to. Thank you.
It’s from Mr Jenkins to you on 6 May 2010, yes?
Anne Chambers: Yes.
Mr Beer: Was subject line of “Receipts payments mismatches”?
Anne Chambers: Yes.
Mr Beer: He says that he’s noticed NT counter events which look like receipts and payments mismatches?
Anne Chambers: Yes.
Mr Beer: Yes? Why was he emailing you?
Anne Chambers: Because I was a useful person who would know what was going on in SSC and could check whether calls had been raised for them.
Mr Beer: Sorry, could check?
Anne Chambers: Whether a PEAK call had been raised for these two instances.
Mr Beer: Why would Mr Jenkins contact you in particular, rather than the other 24?
Anne Chambers: Because I was a helpful person.
Mr Beer: More helpful than anyone else?
Anne Chambers: Probably.
Mr Beer: Okay. He continues “Jon”, and who is that?
Anne Chambers: Jon Hulme, who was, I think, in charge of the counter development team at that point.
Mr Beer: “… that there were also raised from the Office Snapshot erroneously …”
I think should that read “that these were also raised from the Office Snapshot erroneously”?
Anne Chambers: Probably.
Mr Beer: “… but that PEAK [and a number is given] was fixed in [a fixed code] which should be Live.”
Anne Chambers: Yes.
Mr Beer: “Have you been made aware of these or had any calls? I don’t know if there is a KEL for SMC to pick up any such events and raise calls – there certainly ought to be …”
Can you help us, what is an NT counter event?
Anne Chambers: When the counter application would check at various points at the end of the balancing process to make sure that receipts and payments were equal and, if they weren’t, it would flag that in various ways. One of the ways it flagged it was by creating an NT counter event, which would be written to the application event log, which was one of the files we were talking about yesterday.
Actually, no, now we’re on HNG-X, it was very slightly different with the file that had the events in, I think. But anyway, it’s the same sort of thing. And these events would have gone from the counter through the Tivoli stream to be – hopefully to be monitored for and checked by the SMC, whose job was to look for these sort of events or any other unexpected events.
Mr Beer: He, Mr Jenkins, says in his last line there that he doesn’t know if there’s a KEL to pick up such events and raise calls. Now, there was, of course, a KEL.
Anne Chambers: Yes.
Mr Beer: We know that there was the KEL ballantj1759Q?
Anne Chambers: Yes.
Mr Beer: Why would Mr Jenkins not know about a KEL that had been in existence, by my calculations, three months by that time?
Anne Chambers: His job was not support. He didn’t necessarily use the KEL system. He wasn’t responsible for raising them or particularly using them.
Mr Beer: What was the Development team’s access rights to KELs?
Anne Chambers: He wasn’t, strictly speaking, part of the Development team but, yes, the Development team had access to the KELs.
Mr Beer: What was Mr Jenkins’s access rights to the KELs?
Anne Chambers: I don’t know. I can’t now remember if he did have access to them or whether he – it was just easier to ask me, probably.
Mr Beer: He speaks, essentially, of a system being made or needed to raise calls. What’s that a reference to?
Anne Chambers: Well, part of the process of looking out for this type of error was that SMC would – were meant to be monitoring for this type of error, and, if they saw one, then they should raise a call – it wasn’t PowerHelp by then, but whatever it was – which would then get passed on to PEAK for SSC to investigate.
Mr Beer: At the top of the page you reply, copying Mr Parker in. You say:
“Gareth.
“… there is a KEL [then you give the reference] which tells the SMC to raise a call if they see this event.
“I haven’t noticed any calls (but I haven’t been doing that sort of call recently). I do have a PM-raised call from a few weeks back which I need to look at (the mismatch was only for a few pence so it has gone to the back of the heap).”
Was there a heap –
Anne Chambers: Yes.
Mr Beer: – ie a mountain of unresolved systems issues that you had to work your way through?
Anne Chambers: Er, yes, we were very, very busy at this time during the HNG-X pilot. HNG-X was being used at about – I can’t remember if it was 250 or 500 branches and, as you’d expect for any new system, despite having gone through very expensive testing, once you let several hundred branches have a try, they found paths that couldn’t have been gone through during the test process.
So I can’t remember what other sort of call I had been doing but, yes, I had been busy. The postmaster-raised call, I think I say in my witness statement, I shouldn’t have left it that long, even if it was only for a few pence, but it would have been – the effect on the branch wasn’t significant but it definitely needed looking at and it hadn’t just been closed down. It was waiting.
Mr Beer: Were any of these receipts and payments mismatches picked up by the reconciliation process?
Anne Chambers: No, because the events were now being used instead of the reconciliation process for this specific type of error.
Mr Beer: But, on this occasion, it was a postmaster who had raised the mismatch, not the NT events?
Anne Chambers: The call that was on my stack, which I have no memory of now and haven’t had sight of, was raised by the postmaster, yes.
Mr Beer: Was that the case, that even though Fujitsu systems were supposed to pick up things like this, errors were often flagged for the first time by a subpostmaster?
Anne Chambers: Um no, I don’t think that is usual. I mean, I don’t know now whether there had been a SMC-raised call for that call that was on my stack which hadn’t then been linked with it. I haven’t got that information.
Mr Beer: Can we move forwards, please, and look at PEAK PCO203864, which is at FUJ00081586.
If you see, this was a PEAK raised on 2 September 2010 and it concerns a mismatch of a smaller amount of money, £11.20.
Anne Chambers: Yes.
Mr Beer: Yes?
Anne Chambers: Yes.
Mr Beer: Can we turn, please, to page 2 and look at your entry for 18.52.00?
Anne Chambers: Yes.
Mr Beer: You say:
“Joe, this is important because it means that their accounts don’t net to zero due to some sort of system error – not user error. Similar to a receipts and payments mismatch. Garrett had a call about a problem with incomplete summaries recently, worth checking whether that was the same branch.”
Anne Chambers: Yes.
Mr Beer: What are you referring to there?
Anne Chambers: You mean the problem with incomplete summaries?
Mr Beer: Yes.
Anne Chambers: That was this reconciliation report which reported on any branches where the day’s transactions didn’t net to zero. So the branch, if it was the same branch, they might have had that problem one day and then, at some point in the future when they did their balancing then – and produce their branch trading statement, then this situation that this call is about with the trading position not being zero would be reported and I can’t remember if that was on one of the reconciliation reports or if it was an event again.
Mr Beer: Did you think here is a version of the payments and mismatch bug that we saw earlier in the year doing its work again?
Anne Chambers: No, I don’t think so because I thought – we’re missing some evidence here. The earlier problems, we know about the wrongly reported one during the office snapshot. Nobody has shown me the PEAKs that were subsequently raised for those two events that Gareth reported. I am absolutely certain that, him having flagged it up, that would have been followed up on pretty quickly. But we haven’t got those calls for me to look at to give you any explanation of.
So, as far as I was concerned, when I saw this call coming in, I found it alarming. Not because I knew there was already a problem in this area but because it looked like there might be something new.
At this point, September 2010, the rollout of HNG-X to the entire estate was in progress. I’m not sure how far through it had got but now, instead of a few hundred branches, we are now probably onto several thousand branches, with the opportunity to find some new error paths, and so on. So I was obviously concerned that, yeah, we’ve got a problem here and it wasn’t because I knew of existing problems. I thought it was quite likely that there was a new problem.
Mr Beer: Can we go then to the KEL that we looked at earlier, FUJ00081608. Looking at the top, we can see that it was raised by Mr Wright on 23 September 2010 but was last updated by Cheryl Card on 1 April 2016, both SSC members; is that right?
Anne Chambers: Yes, and there have been ten versions of it.
Mr Beer: Yes. We’re looking at the tenth version. It describes the receipts and payments mismatch bug rather well, so if we can just read it together under “Symptoms”:
“When a clerk balancing the stock unit the rollover screen is eventually displayed, and the clerk then presses the Preview or Print button produce the Trial Balance … The counter then returns to the rollover screen.
“Having checked the report, the clerk then presses the Rollover button, and in normal circumstances is given the choice of rolling to a new Balance Period or a new Trading Period.
“If the clerk chooses to roll to a new [Trading Period], the net discrepancies are present, then the system asks whether the clerk wishes to transfer the net discrepancy to local suspense, or else cancel the rollover …
“If the clerk presses Cancel, the system returns to the rollover screen and he/she can press Print or Preview or Rollover or Cancel back to the Stock Balancing menu.”
Then there’s a reference to another KEL.
If we read the solution at the foot of the page. A reference data fix was released in November 2010 under a PEAK, and the number is given:
“Now that the fix has been deployed, if Cancel is pressed on [number given] then the discrepancy is not cleared.
“A Workaround (prior to fix):
“If the Clerk presses Cancel on [message number given], then to avoid the bug they must press Cancel again to return to the Stock Balancing menu.
“Unfortunately the workaround cannot be done after the problem has occurred at the office! In this case the branch accounts will need to be corrected.
“Please advise branches to continue rolling over stock units and the office as normal. It is not necessary to wait for the correction to be applied before rolling into a new TP.”
Anne Chambers: Yes.
Mr Beer: The workaround suggests, is this right, that that was applied in the period before November 2010, before the fix was released?
Anne Chambers: The workaround is really just saying which button the clerk would need to press to avoid the problem. You didn’t read through the problem section on the screen, which is actually where it describes the sequence of button presses that got you into this situation. But, yeah, the workaround was no good unless you were very well aware of what was going to happen.
Mr Beer: So it’s not really a workaround at all, is it?
Anne Chambers: No, no.
Mr Beer: Because it couldn’t be done after the problem had actually occurred?
Anne Chambers: No.
Mr Beer: So it’s not a workaround at all?
Anne Chambers: It’s not a workaround, no.
Mr Beer: That’s because it would always be the case that the problem would come to light after the occurrence in the office?
Anne Chambers: Yes.
Mr Beer: So, is this right: until the fix was applied, Fujitsu were relying on subpostmasters to call in? That was essentially the only step that was being taken?
Anne Chambers: Um –
Mr Beer: There was nothing proactive done?
Anne Chambers: I cannot remember. I wasn’t involved, but I think, in all the various documents that we’ve seen, there was a lot of talk with – between Fujitsu and Post Office as to how to sort this out, to resolve any discrepancies. In fact, in this case, the branch were losing their discrepancies, so they made a loss. This actually lost their loss. If they made a gain, they lost that as well.
But I believe, but it’s in a lot of this other documentation somewhere, that steps were taken by Fujitsu to find all occurrences of this problem and then with Post Office to decide what to do about them.
Mr Beer: What about this: as you rightly said, the problem section of this KEL described a sequence of button presses by a subpostmaster resulting in this receipts and payments mismatch, yes?
Anne Chambers: Yes.
Mr Beer: What about sending a notice out to all subpostmasters saying, “We’ve got a bug in our system, don’t cancel rollovers when you’ve got a discrepancy because it will cause a receipts and payments mismatch”?
Anne Chambers: Yes, um –
Mr Beer: A bit like a sort of product safety recall or a warning notice to everyone that’s using a system, “We’ve got something wrong with our system. Don’t do this, otherwise it will cause an issue”?
Anne Chambers: That would have possibly caused more confusion at 12,000 branches than the problems caused at the – I can’t remember how many it was but I think it was fewer than 100 that were actually affected by the problem. But, yes, that would be something to consider doing. But that would be up to Post Office to communicate to the branches.
Mr Beer: Were you ever aware of such a discussion occurring in relation to this issue, this bug, or any other bug, “Let’s tell people” – relatively simple on this occasion – “don’t cancel rollovers when you’ve got a discrepancy”?
Anne Chambers: That would cause more confusion because they would not want to roll over with a discrepancy that they disagreed with, so you would have to word it very carefully and there was a way of them cancelling – it was just a very specific point that they had to not continue to avoid – or, sorry, not cancel to avoid the problem. It wasn’t the only way they had of backing out to of the process.
Sorry, to get to your question –
Mr Beer: What about the broader issue –
Anne Chambers: Yeah, um –
Mr Beer: – of accepting that there’s a problem with the system and telling the subpostmaster community about it?
Anne Chambers: Yes, I cannot definitely remember. That would not have been up to Fujitsu to make that decision. We had no means of communicating directly with all the subpostmasters. Post Office could send messages that would appear on the screen at the start of day but that was totally within their control as to what they were – wanted to communicate with their postmasters.
Mr Beer: I’m not suggesting, let me be clear, that this should have been something that Fujitsu took on itself to do.
Anne Chambers: Mm.
Mr Beer: It was a service provider to a client. I’m asking whether you were aware in your 16 years of ever any discussion about that occurring, “Rather than correcting things behind the scenes and not telling subpostmasters about them, we actually say there’s a bug in the system”?
Anne Chambers: Um, I can’t remember. I wasn’t usually involved at discussions at that sort of level for problems that would affect a significant number of branches.
Mr Beer: In your witness statement, paragraph 54, you say:
“I am asked whether there were any written or unwritten practices, policies or procedures to restrict what information about a bug or potential bug could or would be shared with others, either for limited periods or indefinitely. I was not aware of any such. If I spoke to a postmaster about a problem and I identified it had been caused by system error, I would say so.”
Again, the revelation to a subpostmaster of a system issue was dependent on you speaking directly to the subpostmaster.
Anne Chambers: Yes.
Mr Beer: I think you told us earlier that that happened very infrequently in your 16 years?
Anne Chambers: No, I said what happened very infrequently was making corrections to the branch financial data. I certainly would have spoken to postmasters most weeks, perhaps not quite as often as that. It would depend on the sort of calls that I was handling, but, yes, it wasn’t that unusual to speak to a postmaster.
It wouldn’t always be to say there was a system problem because sometimes I would be speaking to them for some other reason.
Mr Beer: Was there any guidance or policy on whether or not you should reveal to subpostmasters system faults with the Horizon System?
Anne Chambers: No, I was never given any guidance on that.
Mr Beer: It was a matter of individual discretion for you?
Anne Chambers: Yes, but I and my colleagues certainly would say – I would hear them on the phone talking to postmasters and I’ve seen quite a few PEAKs, and so on, where it says, “Spoke to the postmaster, explained it’s a system problem”. So that was being done.
Mr Beer: Given that you have just said that you did it and you were aware of other colleagues in SSC sitting near or around you doing it, how did that sit with what we discussed earlier: the Post Office’s reluctance to reveal system errors, as you described it?
Anne Chambers: Yes, that seemed to be the policy that they took on some of these bigger issues that were affecting more branches. But within SSC, we were never, ever trying to hide the fact that there were system problems.
Mr Beer: Can we turn, please, to POL00028898. This is PEAK 0204765. You’ll see that it’s opened on 25 September 2010. The summary is, having given the branch code and a message number, “non-zero trading position on office rollover”.
If we look at page 2, please. Scroll down to the entry for 15.16.30, an entry by your colleague Cheryl Card. She says:
“The problem occurred on [15 September] when stock unit 02 rolled over. This was originally reported as per [the KEL that we’ve read] in call [then a PEAK number is given], but for some reason the call was closed without being investigated.
“There is a known problem with the use of the Cancel button during stock unit rollover. This is fully described in [the other KEL we looked at]. A fix is currently being worked on.”
Anne Chambers: Yes.
Mr Beer: Then if we go over the page, you’ll see from the second entry from the top, the call has been assigned to Mr Jenkins on 27 September, for advice on how to correct the branch accounts.
Anne Chambers: Apparently, yes.
Mr Beer: Can you assist, why was this still occurring?
Anne Chambers: Because the fix hadn’t been made yet.
Mr Beer: If we go back to page 1, if we look at the call status at the top, the “Priority” status at the top, it’s described three lines from the top on the right-hand side as “Non-critical”, yes?
Anne Chambers: Yes.
Mr Beer: If a PEAK was given this status, “C – Non-critical”, was that taken into account in a service level agreement with the Post Office when working out penalty clause thresholds of payments by Fujitsu to the Post Office?
Anne Chambers: I don’t know.
Mr Beer: Were you aware of a service level agreement which contained essentially liquidated damages thresholds, depending on the status of calls as between A, B and C?
Anne Chambers: I don’t think so. Only in as far as I said yesterday: I knew that some – priority financial calls did have to be done within certain lengths of time to resolve the financial side of it. But no, I mean this was presumably raised as a C priority by the helpdesk, unless anybody changed the priority subsequently. That didn’t mean that SSC wouldn’t pick it up quickly and investigate it.
Mr Beer: So the priority status didn’t affect the priority with which the SSC dealt with the PEAK?
Anne Chambers: No, not necessarily.
Mr Beer: What was the purpose of attributing a priority status?
Anne Chambers: If it was a priority, it would definitely be looked at quickly, but that doesn’t mean that the rest Cs went to the back of the heap, necessarily. Obviously something with a non-zero trading position would be looked at fairly quickly, I would think. I can’t see how quickly it was given to Cheryl, unless you scroll down.
Mr Beer: I think she first picked it up on the 27th.
Anne Chambers: Right.
Mr Beer: If we scroll to the second page.
Anne Chambers: Yeah.
Mr Beer: I think her first entry is on the 27th. Scroll down, please.
Anne Chambers: That’s so – well, yes. Yes, so – and without knowing which day of the week it is and so on, but yes. So it came into SSC on the 25th and then the investigation started on the 27th, by the look of it.
Mr Beer: If we go to the third page, please, and look at the third entry down:
“The branch accounts will need to be corrected. PEAK [and then a number] has been sent to development for advice as to how to correct the accounts.”
Then do you see there’s some text copied in and, amongst other things, the severity given there is as critical?
Anne Chambers: That’s the severity of the event. These have been – these two entries are from the NT events, which are being monitored centrally.
Mr Beer: Why might an NT event that has been attributed the severity of “critical” be assigned priority status C, of “non-critical”?
Anne Chambers: Because when the call was raised, it wasn’t actually raised for one of these events.
Mr Beer: Would not the fact that this NT event had been recognised to be linked to the call that had been made a cause of recategorisation of priority?
Anne Chambers: It might have made sense for somebody to have increased the priority of this bug from C, but we can see, from all the other documents and calls that were coming in with this problem, there were a lot of people working on it by now. It was not one little C priority call at the back of the heap with people at the helpdesk adding things to it and nobody looking at it. The investigation was well under way.
Mr Beer: I’m going to skip over much of the correspondence, documents and emails from September, October and November 2010 concerning the bug and its revelation to the Post Office, because they mainly concern Mr Jenkins’ actions.
But can we go, please, to a document from mid-November 2010, FUJ00081214. This is a series of emails. Can we start, please, with the third page. Just look at the bottom of the second page to see who it is from and to.
From Antonio Jamasb, a Post Office employee, the branch IT service manager, to Saheed Salawu in Fujitsu. Did you know him or her?
Anne Chambers: Him. He might have been Steve Parker’s boss at this point.
Mr Beer: Sorry, I missed that.
Anne Chambers: He might have been Steve Parker’s boss at that point, but I’m not at all sure.
Mr Beer: If we scroll up a little bit, we can see Mr Salawu’s sign-off block in his signature. So you can see what his role was.
Anne Chambers: So Service Delivery Manager. So I was wrong about him being Steve’s boss.
Mr Beer: Anyway, going back to the text of Mr Jamasb’s email:
“… I have a conference call on Monday with senior stakeholders within POL.
“I need a full update for Receipts and Payments.
“I need:
“Up-to-date spreadsheet of branches affected and what the discrepancy is.
“Up-to-date list of branches/counters yet to have fix.
“Any calls logged with HSD re issue.
“A summary from Fujitsu stating why we have no other integrity issues with Horizon, and why we couldn’t see this issue.
“Sorry to drop this on you.”
In relation to the fourth of Mr Jamasb’s requests or demands, would you agree that that’s a reasonable question for a customer to ask their contractor?
Anne Chambers: It would be a difficult question to answer, possibly. I’ve no idea. I wasn’t involved, as far as I’m aware, in this investigation at the time.
Mr Beer: But would you agree, stepping back, that this is a reasonable reassurance for a customer to seek?
Anne Chambers: I think it’s a reasonable reassurance for a customer to seek.
Mr Beer: Asking “Please tell us, Fujitsu, why we have got no other integrity issues with Horizon?” Part one.
Part two: “Why was it we couldn’t see this issue?”
Can we go to – scrolling up, please, Mr Salawu forwards it to some others within Fujitsu. Can you help who they were, Mike Woolgar and Neneh Lowther?
Anne Chambers: I think they were other what’s obviously called a Service Delivery Manager at this point.
Mr Beer: Second paragraph:
“I know Mike was running with this but there should be information that can answer the queries. It’s a good test of how effective our update process works.”
Then scrolling up still further. Mr Woolgar emails Messrs Simpkins and Jenkins:
“… are you able to provide answers to the questions from POL … yesterday?”
He will deal with the third one and then go to page 1, please. Mr Jenkins reply, he adds Mark Wright in, who I think would have been your manager by then or team leader.
Anne Chambers: He was my team leader, yeah.
Mr Beer: Mr Jenkins says:
“I think it is Mark from SSC that has been running with this rather than John.
“Attached is an email he sent to POL with an update yesterday. I think that addresses points 1 and 2 …
“As for point 4, then that is probably down to me. In simple terms I don’t think we can make such a statement.”
You’ll remember what the request was, “Can Fujitsu tell us why we have no other integrity issues with Horizon and why couldn’t we see this issue?”
He says:
“I don’t think we can make such a statement.”
He continues:
“What we can do is check through what known integrity issues we have and also make the more general statement that when integrity issues arrive, then they do leave a trail enabling them to be identified and their scope to be ascertained.
“John/Mark: are you aware of any other integrity issues we have not yet fixed? I can’t think of any off the top of my head.”
At this time, would you have answered the question in the same way as Mr Jenkins? That you couldn’t say that there are no integrity issues with Horizon?
Anne Chambers: Yes, I don’t think I – I think I would have answered it in the same way. I said earlier I thought it was a very difficult question to answer, and I – yes, I would go along with what he says.
Mr Beer: In relation to what else he said, would you agree that the best that could possibly be said was that there were, in fact, known integrity issues with Horizon?
Anne Chambers: Well, it’s to check through what known integrity issues we have.
Mr Beer: Yes.
Anne Chambers: Yes.
Mr Beer: Would you agree that, when an integrity issue shows itself, it leaves a trail?
Anne Chambers: Yes.
Mr Beer: So you would have answered this in the same way?
Anne Chambers: I think it was way above my pay grade to be answering that type of question.
Mr Beer: So returning to the issues, then, it seems that significant action was taken in relation to the bug in September 2010; is that right?
Anne Chambers: As far as I know. I wasn’t involved.
Mr Beer: You have, I think, answered my question already that PEAKs had been raised from February 2010 onwards and your answer to the question “Why wasn’t action taken in relation to those PEAKs”, was that they concerned a different issue.
Anne Chambers: I think it’s highly likely that they concerned a different issue. We have not seen it, so I cannot say definitively either way.
Mr Beer: But Mr Jenkins appears to have been aware of this bug, a receipts and payments mismatch bug, which caused the Windows NT 902 events from May 2010?
Anne Chambers: No, he was aware of – that there could be various causes of receipts and payments bugs. We haven’t seen anything that links those two that he flagged with the same – with the receipts and payments particular issue caused by the “prev” button.
Mr Beer: In relation to the hundred or so branches that you mentioned earlier –
Anne Chambers: I’ve no idea how many it was. It would be – it’s written down somewhere.
Mr Beer: To your knowledge, what action was taken proactively to tell them of the existence of this bug?
Anne Chambers: I don’t know.
Mr Beer: Thank you. Can we move to the Callendar Square/Falkirk bug.
It might be a good opportunity to take a break and reconvene at 1.50, sir?
Sir Wyn Williams: By all means, yes, that’s fine.
So 1.50, thank you very much.
Mr Beer: Thank you very much.
(12.48 pm)
(The Short Adjournment)
(1.50 pm)
Mr Beer: Sir, good afternoon can you see and hear me?
Sir Wyn Williams: Yes, I can, thank you.
Mr Beer: Thank you very much.
Good afternoon; Mrs Chambers. Can we then turn to the Callendar Square/Falkirk bug, Bug 2. In very simple terms, an explanation of the bug could be as follows, would you agree with it: firstly, it was a big that afflicted Legacy Horizon?
Anne Chambers: Yes.
Mr Beer: It started in about 2000?
Anne Chambers: Yes.
Mr Beer: It was caused by a lock in the Riposte software?
Anne Chambers: Yes.
Mr Beer: You give helpful information in slightly more detail in your witness statement, which I’d ask to be turned up, in paragraphs 73 and 74, which are on pages 23 and 24. In 73 you say:
“Within the SSC we referred to the underlying problem as the Riposte Lock problem.”
That’s instead of the Callendar Square or Falkirk bug?
Anne Chambers: Yes.
Mr Beer: “Normally Riposte messages were automatically replicated between counters so each counter held an identical set of all transaction and reference data relating to that branch. But occasionally one counter would fail to accept any messages from other counters. This usually seemed to be triggered by something early in the declaration or balancing process. Repeated application events were generated which were not visible to the user. The event storm and failure to replicate …”
Just stopping there, what do you mean by “the event storm”?
Anne Chambers: The repeated application events. Every few seconds the same event was generated and we referred to that as a storm.
Mr Beer: “… would persist until the counter was rebooted or ClearDesk was run?”
What was ClearDesk?
Anne Chambers: That was the process that ran at some points in the early hours of the morning to restart the counter application.
Mr Beer: Thank you. Then over the page to 74:
“The counter would still be able to serve customers but would appear to be working normally, but anything done on other counters after the event started would not be visible. Reports printed on the counter would not include transactions done on other counters so those transactions might be re-entered. Incorrect discrepancies could be reported if the money was in the till but the transactions weren’t included in the balance. Transfers between stock units might be accepted in twice, causing a discrepancy and a receipts and payments mismatch. Single counter branches could not have this problem.”
That can come down, thank you. The issues I would like to explore with you, so you know where we’re going, are firstly exploring the explanation for what was done to address the bug in the early 2000s; when Fujitsu was first aware that the Riposte lock could cause a balancing issue; who was aware of that issue; why it was allowed to remain outstanding until 2006; and did the fix, known as S90, work fully.
Okay? So if we can just go through the chronology of those events and try to pick it up as we do.
Anne Chambers: Yes.
Mr Beer: Again, this isn’t a complete chronology, there are about 15 documents that I want to ask you about but there are a very large number of additional documents and steps in the chronology. Can we start, please, with FUJ00017986. This is a PinICL, 00127251, and you’ll see that it was opened on 2 July 1998.
Anne Chambers: Yes.
Mr Beer: The opening summary at the top left-hand side is “Riposte error: Failure to get lock”.
Anne Chambers: Yes.
Mr Beer: I appreciate this before your time by couple to years, in the SSC but, looking through this PinICL, is it right that this appears to be early evidence of the Riposte lock?
Anne Chambers: It is an instance of a single Riposte lock error. There’s no mention in there of repeated events, which were the – it was when you got the repeated events that you might then also have additional problems – it might then affect the replication. One single event, we never had any evidence that that caused any long-term problems.
I’d also say that I don’t believe this is anything to do with the counter software. I don’t think it was this process. “B_LD_CD_DEL” looks like one of the bulk loaders that would have been running on one of the Horizon back-end systems.
Mr Beer: So following from that, if we look at the last few entries on page 4, if we look at six lines up, Mr Bell’s entry:
“I have not seen this problem since the test rig was updated to Riposte 216.
“Also the network has been changed so I’m closing this call.”
Does that tend to suggest that because the problem was not seen as at 5 October 1998, the PinICL was therefore closed as an isolated example?
Anne Chambers: Yes, it was an example of a single Riposte error which – I can’t tell from this but there’s no evidence that this one single error – and, you know, you do get errors and your systems have to cope with this. There may be a good reason why it failed to get the lock and it reported it. So it’s an instance of that particular error message, and, yes, they didn’t see any more of it on the test rig, so they closed the call.
Mr Beer: When you and other SSC staff, much later or many years later on, were investigating the extent of the Riposte lock bug and the duration of it, would this kind of PinICL have been available to you?
Anne Chambers: I don’t recall how long PinICLs were kept for. It might have been there but, looking at that, it bears absolutely no relevance to a counter balancing problem found some years later.
Mr Beer: So if you had had access to it, you would have dismissed it as irrelevant?
Anne Chambers: Yeah, as –
Mr Beer: Can we move forwards then, please, to FUJ00031913. We’re now on 5 November 1999, so again before your time, but it’s another PinICL opened concerning another Riposte lock, yes?
Anne Chambers: Yes, it is, which happened at a particular time of day for a particular process.
Mr Beer: Reading through the PinICL, would you agree that it appears that the SSC took no substantive action in relation to the lock and, instead, it was simply closed on 11 November 1999?
Anne Chambers: Assuming it’s closed further down, it was something – ClearDesk failed to create training object, that is the overnight processing starting things up again, it’s trying to create something to do with the separate training service that ran on the counter. It got, it looks like, a single timeout message and also this “error occurred” message. It doesn’t show any lasting ongoing problem. It wouldn’t affect replication in any way, so if it was just closed, I’m not surprised.
Mr Beer: You’re not surprised it was just open and closed very quickly?
Anne Chambers: Yeah.
Mr Beer: Can we move forwards, then, to the year 2000, FUJ00059049. You were in post by this time?
Anne Chambers: I’d been there about three weeks, I think.
Mr Beer: Yes. We’ll see that this is a KEL raised by Mr Ballantyne on 2 November and then closed by you in 2005?
Anne Chambers: Not closed: last updated.
Mr Beer: Sorry, last updated by you. If we see the “Solution”, please, further down the page.
Anne Chambers: Yes, could I just say, sorry, before we go down, this – where they’re getting the error messages committing the discrepancies, and so on, this suggests that the underlying problem which is happening, it’s not just preventing the replication between counters; it’s actually a problem on this counter itself, where Riposte is trying to write messages into the message store on this particular counter and it’s not able to do so.
Mr Beer: So I’m not following for the moment what the point is or the distinction you’re making.
Anne Chambers: I think when I described the problem in my witness statement I said the problem was the failure to replicate messages from – that were being done on one counter onto this counter that they were balancing on.
This particular description, which I don’t think I’d had the opportunity to remind myself of when I wrote that, it’s clear that they are doing the work on this one counter and then, when, in the balancing process, it’s trying to write the declarations and the discrepancies into the message store, it’s unable to do that, presumably because this lock is held and so it can’t write into its own message store.
Mr Beer: So this could afflict a single counter; is that what you’re saying?
Anne Chambers: It possibly could, yes, but because it can’t commit or do any of these things, you’re actually not going to be able to complete your balance process on the counter in the state that it’s in at the moment anyway. But yes, it – reading it now, I think it might affect a single counter.
Mr Beer: I see. In terms of the solution, just reading the “Solution” to yourself, is it right that in essence the solution was to advise a restart by the subpostmaster and to stop balancing if they were doing the balance?
Anne Chambers: That is the solution that was being given at that time, yes.
Mr Beer: How long was that solution to stop balancing and restart?
Anne Chambers: That was for a long time because it took until 2006, I think, for us actually to get a fix for the problem.
Mr Beer: So was that the operative advice for about a six-year period?
Anne Chambers: Yes, it would have been.
Mr Beer: Can we look, please, at the PEAK to which this KEL is associated, 0056922, and the PEAK is FUJ00070841. Now, again, you’re in post by this time but for a very short period of time. I’m not showing you the PEAK because you’re mentioned in it. It’s another in the line of documents evidencing the existence of the bug.
Can we look at page 4, please, at the foot of the page, and Mark Jarosz’s entry at 10.35.00, thank you. He says:
“My assessment of what happened is that on Wednesday 1st Nov at 18.32.13 a lock was acquired on the run table which was not released. This had the subsequent effect of causing [I think that’s ‘many’] Riposte API calls to fail and hence the applications connected to Riposte could not function reliably. I would speculate that the probable cause was a thread silently failing but we have no way of proving this.
“I will check with Escher to confirm my assessment is reasonable and if not further update this PinICL.
“In the meantime I would recommend that in future occurrences a restart of Riposte should be attempted prior to rebooting NT.
“If the frequency of occurrence of such an event becomes significant ([more than] 1 per month) then we will need to create a reproducible case.”
Can you explain what you understand Mr Jarosz to be saying there?
Anne Chambers: Jarosz. Some process has failed but it’s not being picked up, he doesn’t know what that will be. I don’t think I can explain that very clearly, technically. He was going to check with Escher, who owned the Riposte code, because this was the underlying product that Horizon was built around, which Fujitsu didn’t support, and –
Mr Beer: The last point I’m interested in particularly: if the frequency of occurrence becomes significant, which he defines as meaning more than once per month, “we’ll need to create a reproducible case”.
Anne Chambers: Yes, so if it keeps happening more than once a month, then we’re going to have to see if we can reproduce the problem, which actually we never managed to do.
Mr Beer: Who would monitor whether such incidents, concerning the Riposte lock, were occurring at a rate of greater than or less than once per month?
Anne Chambers: I’m not sure that anybody was monitoring that at that point.
Mr Beer: How would the proposal or conclusion or outcome that Mr Jarosz has arrived at there be carried into effect then, or wouldn’t it?
Anne Chambers: I don’t know.
Mr Beer: If nobody was monitoring the frequency with which such events occurred, it couldn’t be, could it?
Anne Chambers: No, it couldn’t. I don’t know if any of my colleagues back then, who were aware of this problem, took it upon themselves to do such monitoring but I certainly didn’t do it, because I hadn’t been there very long and wasn’t really aware of this problem at that point.
Mr Beer: More generally, was there a system within the SSC of logging disparate PinICLs and PEAKs together so that some sort of meta analysis could be carried out?
Anne Chambers: Not to my knowledge. I don’t know if that’s something that the SSC manager did have any – did have any systems in place for, but I’m not aware of that.
Mr Beer: Were you ever required or requested to contribute to such a system, either in its design or providing data to it?
Anne Chambers: There certainly were occasions when I would do my own checks for similar calls happening. But I don’t recall it being something that I was ever instructed to do.
Mr Beer: So there was nothing to stop one PEAK coming in to one person in the SSC and another coming in to one of the other 24 people in the SSC about the same subject matter, and person 1 not knowing about person 2’s PEAK?
Anne Chambers: That could happen. I mean, what we did sometimes do is, on the KELs say “Record further instances here”, so then we did get a bigger picture but that wasn’t part of the process that anybody told us to do.
Mr Beer: It was a bit hit and miss, I think it’s fair to say?
Anne Chambers: It could be hit and miss, yes.
Mr Beer: So when one looks at the KELs, one doesn’t see a list of all the associated PEAKs?
Anne Chambers: You would on some KELs for particular problems.
Mr Beer: But not on many others?
Anne Chambers: Not on many others.
Mr Beer: Were you conscious of this within the SSC at the time, thinking “I’m working my way through my heap, my stack of tickets, I’m getting them in, dealing with them, getting them out, and there could be somebody else who’s working a different shift to me, somebody home working”, I don’t know – probably not home working in the SSC –
Anne Chambers: (Unclear).
Mr Beer: – but working a different shift, working on the other side of the room, “and they could be exploring precisely the same problem and we don’t know about it”?
Anne Chambers: That is possible. Certainly if it was the same – at the same point in time, we’d almost certainly notice just because you would be keeping an eye on the other calls that were open and you would see if somebody had a similar call, or the pre-scanner would say, “Ooh, I’ve just given one that looks like this to so and so”, but if they were, you know, several weeks apart you would not necessarily make those links.
Having said that, in – certainly, we did talk to each other, and so we often did have a pretty good idea of other things that were happening so you would get some sort of an idea of, you know, “Oh, there’s another one of those sort of problems”, but it wasn’t being formally measured or managed.
Mr Beer: There was no system in place?
Anne Chambers: No.
Mr Beer: Can we look, please, at one of your KELs at POL00030325. This was a KEL, “AChambers330S”, raised by you about a month into your time, yes?
Anne Chambers: Yes.
Mr Beer: On 27 November 2006, last updated by Mike Croshaw on 20 October 2006. If you scroll down, please, looking at, without reading them out, the “Symptoms” and the “Problem”, this looks like another similar example of the Callendar Square bug, doesn’t it?
Anne Chambers: The KEL was originally raised for – very specifically just for a single occurrence of the event at particular point in processing during the LFS end of day processing, when it wouldn’t have affected – that’s not part of the counter balancing process. Where it goes on to say “sometimes a storm of these events occurs”, that that is later what we have called the Callendar Square bug.
Mr Beer: The Riposte lock?
Anne Chambers: Yes.
Mr Beer: It said under “Solution” that:
“A single event can be ignored.
“Do not pass further instances to SSC unless there appear to be side effects.”
Why was the KEL signed off in this way?
Anne Chambers: Because a single event is just “Oh dear, Riposte has had a slight problem, it’s obviously recovered from it”, in that we only have the one event, we haven’t got ongoing problems. So if there don’t appear to be any side effects, then it doesn’t need any further investigation.
Mr Beer: Was there a concern that too many issues were being sent up to the SSC?
Anne Chambers: I’m not aware of that particular statement in that particular KEL causing that particular problem.
Mr Beer: But the KEL is meant to discourage, isn’t it, passing instances up to the SSC?
Anne Chambers: If just single event has occurred.
Mr Beer: Wouldn’t you want to know where single events had occurred if they were occurring as single events across the estate?
Anne Chambers: Not if they – um …
Mr Beer: Wouldn’t that help identify the problem?
Anne Chambers: It might have done if there was a problem caused by these events being raised. I realise we’re getting into a state where – a chicken and egg situation here. But, yes –
Mr Beer: I’m going to ask you about the chicken and egg situation in a moment.
Anne Chambers: Yes, I certainly wasn’t trying to – I don’t believe that KEL has been written in that way to necessarily stop anything being sent to SSC for any of these events. It was more written in the first place for the single event, I believe.
Mr Beer: But this is addressed to HSH, isn’t it, the solution, to the Helpdesk?
Anne Chambers: Yes, it is, saying a single event can be ignored.
Mr Beer: Was there sufficient skill and expertise within the Helpdesk team to identify whether or not a call related to a single event or was, in fact, one of a series of events?
Anne Chambers: You could –
Mr Beer: How would they know?
Anne Chambers: Because when – if HSD or SMC were monitoring the events, they would see each event coming in as a separate entity from a specific branch.
Mr Beer: What a branch calling in?
Anne Chambers: No, this isn’t a branch ‘phoning in, this is the automatic feed they get through Tivoli of the events, the NT events that are being raised on the counter.
Mr Beer: How would those lower levels of support identify if there were what is described as “side effects”?
Anne Chambers: Um, if further processes started raising other events, if we’re talking about the events, you know, you might get one event saying, you know, that a lock is held, and then other processes might then generate events because they couldn’t do what they were meant to be doing. I mean a lot of events were being raised all the time from a lot of different processes, not that many critical events. There were different categories of events. But we certainly didn’t expect every single event being raised by the system always to be individually investigated.
Mr Beer: Whether or not it required to be individually investigated or not, wouldn’t it be important knowledge for the SSC to have, as to the existence of these individual events, as you called them?
Anne Chambers: Quite honestly, I don’t think we would have been able to do anything about them, except to look at it and say “Well, we can’t see that it’s caused any knock-on event on any other counter process”. The only thing that perhaps it might then have had to go off to Escher again to say, “Can you investigate why these are getting these?” But you do get unexpected errors happening on systems. Systems have to be written in a way where they can cope with unexpected errors at this sort of level.
Mr Beer: Some of the errors were causing what you’ve described as side effects and some were not.
Anne Chambers: Um, I’m not sure if we ever – I’m – yes, it’s hard, very hard now, to go back and say – you know, if we had investigated a single event, we would have had to have looked at the bigger picture of which other processes had been impacted. Certainly this one, just after midnight, during LFS end-of-day processing, it – I think that one possibly did affect some counters that night, sorry, but not the balancing just the LFS process, which didn’t affect branch balancing in any way. I don’t think I can add much more here.
Mr Beer: Okay, can we look at version 2 of this KEL, please, which is FUJ00059141?
Anne Chambers: Which version was this that we were looking at?
Mr Beer: This is version 1 we’re looking at the moment. If we scroll up, we can see the top of it. Now we’re looking at version 2.
Anne Chambers: Right, so that.
Mr Beer: Scroll up. That’s version 1.
Anne Chambers: Version 1 must have been the update by Mike Croshaw in 2006.
Mr Beer: Yes.
Anne Chambers: Which is confusing but the KEL system changed I think at that point and that’s why it went back to version 1, so there would have been earlier versions as well. I think.
Mr Beer: Okay, I’m not going to explore that any further. Can we go to version 2 then. Again, it shows correctly the first date that you raised this KEL, 27 November 2000, and we can see that this version 2 is last updated by your manager Mr Parker on 14 June 2010. Can we take from that that this is confirmation that these lock-type problems were continuing to be experienced throughout the time that Riposte was in operation?
Anne Chambers: Firstly, I’d say I’m wondering if that’s some sort of administrative update, given that we were just about to go live on to HNG-X in 2010 at that point, and go fully live.
Um, I think we very occasionally did continue to see just a single event at odd times, but not the event storms that were happening during the balancing process, which we saw up to 2006.
Mr Beer: Again, was there any method by which that data, from which I’ve called a meta analysis, could be conducted, was retained?
Anne Chambers: Well, all the events were retained, somewhere, for some length of time.
Mr Beer: Yes. In a system but not being looked at.
Anne Chambers: Yes.
Mr Beer: I’m asking whether anyone – where a bug was identified like this, whether anyone in SSC or elsewhere, for example in order to go to Escher to say, “Look this is a continuing problem”, retained dataset that they could go to Escher with and say, “This is a continuing problem it’s been going on for 10 years”.
Anne Chambers: I know that in 2006, after the fix for the event storms went live, I did monitor for some time after that to see if the event storms stopped and the event storms did stop, but there may have been the very, very occasional single event still happening. But it was the storms that seemed to cause the message store either not to replicate or it not to be possible to write to them. Otherwise, if you’ve just got a single problem the processes would retry and it would work on the retry.
Mr Beer: Can we turn, please, to FUJ00083548. This is an email exchange between Mr Jenkins and Mark Jarosz and Brian Orzel. If we just read through the message at the top of the page together, from Mr Jenkins:
“I’ve had a look at this event log and I don’t think there’s anything to really worry about. Migration appears to have completed OK and the outlet is running fine on CI4.”
Was that a release, CI4?
Anne Chambers: Yes, I believe so.
Mr Beer: “I’ve seen number of ‘verification failures’ during migration before and I believe that they are to be expected during the various loads of Riposte before the counter reboot.
“However, I’m curious as to why we get the three errors mentioned in the PinICL. They occur at 20.26 on 9/11/00. All are identical: Facility MessageProcessor … Error 94 ‘An error occurred while attempting to destroy a checkpoint run. Timeout occurred waiting for lock’. They seem to occur during the Riposte index rebuild immediately after the migration of the ‘real’ message store. I assume that they are benign, but would appreciate confirmation from Mark before closing the PinICL.”
I’m not going to look at the rest of the message. For this kind of thing that Mr Jenkins was doing, would you know that this kind of thing was going on, ie the things that he is describing?
Anne Chambers: Well, he’s referring to a PEAK, 57478, and mentioning three identical errors. I don’t think that’s the PEAK that we were looking at before.
Mr Beer: It’s not, no. It’s a separate one entirely.
Anne Chambers: Right, so I think this is a different problem that we’re looking at here. But a different problem but, yes, again, you’ve got the same underlying –
Mr Beer: The same underlying cause?
Anne Chambers: – event –
Mr Beer: Yes.
Anne Chambers: No, not the same – not necessarily the same underlying cause but the same events have been generated and in this time it’s during the phase of the index rebuild, and this is looking like it’s part of the migration. So when branches are first moving on to Horizon. So, yes, some Riposte errors were looked at.
Mr Beer: Can we move on, please, to FUJ00083574. Look at the message in the middle of the page, again from Mr Jenkins. Different PinICL, 57957. He says:
“This PinICL is related to 56922 which you looked at couple of weeks ago.”
That’s not the one that we were just looking at. This is a third one.
Anne Chambers: This looks like we’re back to a problem in the LFS space, which is what that KEL of mine was talking about.
Mr Beer: Yes, and he says:
“I’ve had a look through the message store and the event log and have noticed that at the time of this failure [12.02 am, essentially] that there is an LFS background task running.”
Anne Chambers: Mm.
Mr Beer: He says, next paragraph:
“I suspect that it is significant that the Riposte error is 10 secs after the BLOB is written …”
What was the BLOB?
Anne Chambers: I think it was a Binary Large Object.
Mr Beer: What does that mean?
Anne Chambers: A big amount of data that was so big that it had to be written into a whole set of messages. It was too big to fit into one individual message.
Mr Beer: He then says, next paragraph:
“As the PinICL says, this seems to be happening fairly frequently.”
Yes?
Anne Chambers: That’s what it says, yes.
Mr Beer: Next paragraph:
“I do think we need a definitive statement from Drew …”
Do you know who Drew was?
Anne Chambers: No.
Mr Beer: “… as to whether this event is benign or what problems we could have when it happens. Could it be due to an application error? Do we need to get more info on when these problems occur. It is clear that the circumstances in this case are very different from those in the original PinICL.”
To your knowledge at this time, we’re late November 2000, was this problem happening fairly frequently?
Anne Chambers: I don’t know. I would say again, this isn’t the problem with the repeated events that was affecting balancing.
Mr Beer: The reply at the top of the page, Mark Jarosz says:
“From your description [Gareth] it sounds as though we potentially have a recipe for a reproducible case.”
Can you assist us with what a reproducible case is?
Anne Chambers: One where you could try the same process and reproduce and make the problem happen in a consistent way.
Mr Beer: What would be the purpose of forcing the problem to occur?
Anne Chambers: Well, if you can reproduce it then you stand a much better chance of, firstly, finding the root cause and, secondly, being able to test to show that you have fixed the problem.
Mr Beer: Can we move forwards, please, to FUJ00083583. We can see Mark Jarosz’s reply to Mr Jenkins of, I think, 1 December 2000. It must be an Americanised date, I think.
“Hi Gareth,
“I can confirm (having checked with Drew) that a timeout of this sort is likely to be benign in the sense that it should not result in a message store corruption.”
“At this stage, can you remember whether you in the SSC were told about this investigation having taken place?
Anne Chambers: I have no recollection of this and, again, we’re still talking about problems in the area of the LFS agent, which is nothing to do with counter balancing.
Mr Beer: So at the moment would you draw a distinction to say this isn’t really directly relevant to the bug that we’re looking at?
Anne Chambers: Yes.
Mr Beer: Therefore, you didn’t need to know about it?
Anne Chambers: Personally, probably not, no. There was an awful lot to learn about when I first started, as you can imagine.
Mr Beer: Yes. Can we move forward a couple of years then, please, and look at FUJ00083633. You’ll see that this is a PEAK, 0083563 opened on 7 November 2002 and can we look at your entry, please, at 16.27.00, towards the bottom of the page.
You say:
“Following critical event generated on various FADs …”
That’s by various branches, is that right, or about various branches?
Anne Chambers: Yes, at various branches, not raised by various branches.
Mr Beer: Yes, about various branches, thank you:
“The call summary is now:
“Many ‘run map’ critical events on various [branches].
“Response:
“These events were investigated in the past …”
You give a PEAK number.
“But the call was closed on the basis that the errors were no longer occurring.
“Analysis of the events in the last month shows 2,132 of these events. In many cases there is just one, or a small number on effective counters, but [and then you give a branch number] generated over 900 in one day, and 191323 over 100 [presumably 100 days] …”
Anne Chambers: No, that would be 100 events.
Mr Beer: Over 100 events?
Anne Chambers: Sorry, that’s – the “191323” is a branch code, a FAD code.
Mr Beer: I’ve got it. So there are two branches you’re referring to?
Anne Chambers: Yes.
Mr Beer: The first one generated over 900 in one day?
Anne Chambers: Yeah.
Mr Beer: The second branch over 100 in one day?
Anne Chambers: Yes.
Mr Beer: Thank you. I understand.
Then if you look down the rest of the PEAK, the trail seems to go cold, nothing happens.
Anne Chambers: Yes, I mean, I asked for the call to be raised so that I could do some background investigation into these events, which I had noticed, and I was concerned about, because they were happening. But there was a call that I sent to Development or was sent to Development at around the same time for them to investigate.
Mr Beer: But there appears to be a four-month gap between your entries of 7 and 8 November, and your entry of 24 February?
Anne Chambers: Yes. It was something I was doing as a background task because I was concerned about the events.
Mr Beer: But what was happening in the meantime to these two branches that you mention here?
Anne Chambers: I can’t remember now if I looked – I mean, the branch – the events would only have – they had happened on those particular days. The events didn’t keep having the same – they didn’t keep affecting the same branches over and over again, and whether I did look back to see if the events had had any impact on the branches on those days or whether they had raised calls to say they were having issues, I now cannot remember.
Mr Beer: But if you had have done that you would have noted it on this PEAK, wouldn’t you?
Anne Chambers: Um –
Mr Beer: The branch you’d had at 900 events in one day and the other branch you’d had over 100 events in one day?
Anne Chambers: Um, yes. I mean these events didn’t necessarily mean that they did have balancing problems. It just meant they could, in the majority of cases. It didn’t –
Mr Beer: You don’t know until you look, do you?
Anne Chambers: That’s true. And I don’t – I cannot say now whether I did look at those at that point. It may be that, you know, at this point – I’m not sure when we first realised that it was tending to happen more when they were doing their balancing and therefore it might have an effect, but, yes, it obviously was important to look and, as I say in my witness statement, I’m not at all happy about how this was handled over the years.
Mr Beer: When you say that, you mean by Fujitsu?
Anne Chambers: Yes, by Fujitsu, by SSC. By all of us. We could have done better.
Mr Beer: In relation to your part in that, you made a record on 7 November that, in the last month, there were 2,132 of these events and you highlighted two branches, one where 900 events had happened in one day and another where over 100 events had happened. If you had investigated whether any of those events had caused discrepancies, you would have noted it down on this PEAK, wouldn’t you?
Anne Chambers: Yes, if they had been aware that there were discrepancies, that they had persisted through the event storm and had managed to balance and it looked as if they did have discrepancies as a result, then I probably would have recorded it.
If I had looked and seen that either they’d realised they’d had a problem, had phoned for advice and had rebooted and then done their balancing, I probably wouldn’t have recorded that. But since I didn’t record anything, I don’t know.
Mr Beer: Were you essentially relying on subpostmasters to spot a problem and call it in?
Anne Chambers: I certainly think at this time, with this problem, we did assume that they would notice that – either that they were getting error messages, as we saw in one of the PEAKs you just showed me, where they got error messages because they couldn’t commit the declarations or whatever else. So they’d definitely know they had a problem in that case.
In other cases, we certainly did get some where they phoned in and they said “We’re balancing, our figures are all over the place”, and then the Helpdesk would advise them to reboot and then it would be okay when they restarted.
Mr Beer: Can we move forward again, please, to look at FUJ00083651. This is a PEAK, 0086212, opened on 24 January 2003. Can we look, please, at your entry on 29 January at 11.31, which is on the bottom half of the page, tab. You say:
“It looks as if there was a problem with last week’s balance – cutoffs and some final OBCS transactions were done on counter 3., then balancing continued on counter 4, but this did not seem to know what had been done on counter 3 (there were many underlying Riposte timeout messages). The transactions were ended again; I need to ascertain whether they were sent to TIP twice and whether the postmaster is out of pocket.
“Have spoken to [postmaster] who confirms there were problems and is worried that they may continue this week. I’ll contact her tomorrow am to see how they have got on.”
So there you’re essentially describing, in a single paragraph, the operation of the Callendar Square bug, as I have summarised it, in existence on 29 January 2003.
Anne Chambers: Yes.
Mr Beer: Then if we can look at your entry a couple of days on, 31 January at 16.09, you say:
“[Postmaster] balanced okay. She has reversed the transactions which she had had to reenter (the original ones were included in the new CAP). This was all caused by counter 4 being unable to see messages recently written on counter 3 when the stock unit was being balanced.
“There is no accounting discrepancy here, but there is a problem in that the [postmaster] was allowed to balance with no warning that the counters weren’t communicating. MSU informed that I’m sending this to [Development] for further investigation.”
Anne Chambers: Yes.
Mr Beer: So would you have informed the MSU here?
Anne Chambers: Yes, I would have informed MSU because this call was not raised by the postmaster but because of an entry on one of the reconciliation reports. At this point in time, the cash account was – so when the branch balanced, they did their balance reports and then they produced a cash account on the counter, but the cash account was also reconstructed at the data centre from the transactions that had been harvested – had reached the data centre, and a comparison was done, line by line, to make sure the two were in step. And in this case they weren’t because the data centre, when it did its recalculation, knew about these counter 3 transactions, which hadn’t been picked up when the branch cash account was produced.
So there was a mismatch, and a reconciliation call was raised to investigate why that had happened.
So in this case, that’s how we knew about the problem. I did phone the branch to see what had happened, you know, whether they had realised there was a problem. They had put these transactions in again, because they didn’t think they’d had them once, but then, because the original transactions hadn’t been included in the accounts for the period that they’d just balanced, they automatically got carried forward and then were picked up in the new period.
So, in order to avoid them going through twice, she was then able to do a reversal on them, which sorted out her branch accounts. But obviously, yes, there was still a system problem.
Mr Beer: That’s why you sent it to Development for further investigation?
Anne Chambers: So I sent it to Development for further investigation.
Mr Beer: If we just go forward to complete the story on this PEAK to page 4. Two years on, there’s a record at the top of the page, for 5 October 2005:
“This call is one of a set approved by … (Mik Peach) for closure without further action.”
Was that because the fix was then thought to be the S90 release?
Anne Chambers: I doubt it. I don’t know. I’m not very happy with that.
Mr Beer: Do you know –
Anne Chambers: Was there anything above that at all to say if anybody had looked at it?
Mr Beer: You can look back, please, at page 2. I don’t think there’s anything relevant in your entry on 3 February.
Anne Chambers: No.
Mr Beer: If you scroll to the bottom half of the page, you’ll see two customer calls. If you just read those.
Anne Chambers: Oh, I think that just happened automatically when – was that when we moved from PinICL to PEAK, or something changed and all the calls had to be closed and everyone reopened and then everything got written in again.
Mr Beer: So nothing of substance there?
Anne Chambers: No nothing of substance.
Mr Beer: Then on to page 3?
Anne Chambers: No –
Mr Beer: Nothing there?
Anne Chambers: Yes, nothing there.
Mr Beer: Then bottom half of the page.
Anne Chambers: No.
Mr Beer: Nothing there?
Anne Chambers: No.
Mr Beer: It’s just closed off, isn’t it?
Anne Chambers: It was just closed off.
Mr Beer: Why are you unhappy or not very happy?
Anne Chambers: I’m unhappy with myself because I should have made it something more than a C priority. I left it at the priority that it had come from MSU in the first place, and I should have shouted a lot louder about the fact that this needed looking at. As time went by, I got better at shouting louder.
Mr Beer: Who would you shout to?
Anne Chambers: Oh, anybody who’d listen.
Mr Beer: Meaning who: Mr Peach?
Anne Chambers: Yeah, Mr Peach, Development team.
Mr Beer: So, in essence, this call is closed off without anything having been done on it for two years?
Anne Chambers: Yes.
Mr Beer: One way of describing that is suboptimal, isn’t it?
Anne Chambers: You could say that.
Mr Beer: Can we look at a different PEAK, please. POL00000996, this is PEAK 0103864, opened on 3 June 2004. The “Summary”:
“[Postmaster] reports that he had a problem with some transfers.”
Anne Chambers: Mm-hm.
Mr Beer: I think, without going into the detail, the PEAK describes the problem where multiple transfers in occurred to a stock unit?
Anne Chambers: Yes.
Mr Beer: Can we go forward to page 6, please. In the middle of the page, 6 July at 11.47.27, you’ve written:
“I’ve checked with Mike King; the BIMS report for this problem was sent to POL on [22 June] and should have result in an error notice being sent to the branch. Mike says he will send a note to POL saying that the [postmaster] has been changing this issue; I’ve asked [the helpdesk] to inform the [postmaster] that they should have received an error notice and to check with the department that issues them.
“The corrected cash account that was sent still had a [receipts and payments] mismatch. The double Transfer In causes a mismatch both because of the transfer and because of the discrepancy which has been erroneously generated. The host-calculated [cash account] ignores the transfer but is still affected by the accepted discrepancy which should not have been generated. It is not really possible to provide a fully balanced [cash account] …”
There’s an email on this subject.
Anne Chambers: I think we discussed this one yesterday.
Mr Beer: Yes. You’re recorded as dealing with the BIMS here on 6 July 2004. Can you explain, please, again, exactly what you’re doing there?
Anne Chambers: Yes. I wasn’t personally involved with the BIMS. I checked with Mike, who was the person who had sent the report to Post Office about the discrepancy that shouldn’t have happened at this branch.
Mr Beer: So are you just identifying that there’s been a delay here?
Anne Chambers: Yes, I think so. I think further up the call, Catherine had been dealing with it. She had sent the information to Mike in MSU informing Post Office of this discrepancy that the branch should not be held liable for. The branch had not held anything. They were chasing it back, so the call ended up with me, so I followed it up, as best I could, with Mike.
Mr Beer: Again, could we go forwards in the tale, please, to 2005 – sorry, 2006, and look at POL00030241.
This is a chain of email correspondence on Callendar Square itself –
Anne Chambers: Yes.
Mr Beer: – once the bug had seemingly been identified and discovered within the Post Office. We should just set the context by starting at the foot of page 9 and on to page 10, with an email from Sandra MacKay of POL, to Brian Potter of POL. This is just to set the context for what happens later in the chain. Can you see – I’ve said that’s to Brian Potter. If we just go up, it’s Shaun Turner copied to Brian Potter. Just trying to work out – ah, yes.
Shaun says:
“Gary,
“Need your advice on this branch. There appears to be an ongoing problem at this branch with transfers between [stock units] causing a receipts and payments mismatch. This first came to my attention some 3 or 4 months ago, when the branch was chasing up an error notice to account for a loss that they had that was related to the [receipts and payments] mismatch. I believe in that case, that FS [Fujitsu, I think] had taken it on board and were investigating it as a problem (I seem to recall it had a PinICL number). I had to do some chasing around with P&BA [Products and Branch Accounting] to ensure that the error notice got issued, as there was a breakdown in processes between them and FS relating to the BIM report.
“Since then it appears to have happened again, although Fujitsu are saying no issue could be detected. I am concerned that there is a fundamental flaw with the branches configuration, and would be interested to know how [Fujitsu] put the … issue to bed.”
If we go further up on page 9 and on to page 8, and if we just scroll gently up, we can see that this gets passed around essentially within POL, yes?
Anne Chambers: Liz Evans-Jones was Fujitsu.
Mr Beer: So it goes over to Liz Evans-Jones from 15 February 2006, from Gary Blackburn:
“Liz
“I have had the incident below forwarded to myself by our Service Line … could you please update me on the corrective action plan as this still appears to be occurring within the branch.”
Then if we go further up to page 8 Ms Evans-Jones replies:
“I have checked the call and this issue appears to have been resolved in S90.”
Could you, in a word or two, explain what S90 was?
Anne Chambers: It was a fairly major release of updated software. I can’t remember what functionality it included, but there were new areas of functionality coming in fairly frequently and so, as a part of that, there would be some bug fixes and this was scheduled to be one of them.
Mr Beer: So a scheduled software release?
Anne Chambers: A scheduled major software release, which would have been through a very thorough test cycle.
Mr Beer: It was proposed to include this fix within S90?
Anne Chambers: Yes.
Mr Beer: “S90 has already been deployed to the Datacentre and counter release is scheduled to start”, and then there are some dates:
“3rd line support has been discussing with the [postmaster], and the last contact with the branch (according to PowerHelp) was on 1st Feb. The call has been left open for 3rd line to check to see if the issue reoccurs S90.
Please let me know if I can provide any other assistance …”
Then continue scrolling up, please, and again, please. We can then see some passing, essentially back down the chain, of Liz Evans-Jones’ reply within POL, yes?
Anne Chambers: Yes.
Mr Beer: On page 7, keep scrolling up. Thank you. If we then just scroll down a bit so we can see it is Mr Turner asking these questions. Thank you. Shaun Turner, yes, and then just scroll back up. Thank you.
He says, back to Mr Blackburn:
“Gary,
“Thanks for looking into this for us. Couple of questions occur:
“Do we understand why this particular branch has been having problems? Or are there other branches in the network that have been having this problem?
“Could this branch be front ended on the counted release of S90 such that it gets the fix as soon as possible?
“The email from Liz suggests that there may be a reoccurrence following S90. What degree of certainty do we have that it will definitely be fixed?”
So some pretty direct and pertinent questions there from Mr Turner, yes?
Anne Chambers: Yes.
“Sandra/Brian – Appreciate this is frustrating for the branch but from the email below you can see that the branch’s issue should be fixed for the release of the S90 software. I have asked Gary above to see if we can put this branch to the front of the queue … In the meantime it is important that the branch continues to report any issues into [the Helpdesk].”
So the four or five rather pertinent questions that Mr Turner asks, if we scroll up, please. We then see these getting passed around within Fujitsu. Keep scrolling, please. Keep scrolling – keep scrolling.
We can see an email from Mike Stewart to Mr Simpkins, copied to you. Why were you copied in?
Anne Chambers: I think because Mike says below, “As Anne is away could I have your comments as you were involved as well.”
Mr Beer: That’s saying why John is asked the questions, but why is that addressed to you?
Anne Chambers: Well, before that, it’s “Anne, you’re always a good place to start”, so it was me being a good place to start.
Mr Beer: I’m so sorry, why were you the good place to start?
Anne Chambers: Because I knew what it was going on and because I had to put an update on that call there that’s at the very bottom of the screen, so I’d obviously had some involvement –
Mr Beer: With this bug?
Anne Chambers: – with this bug, as far as I can see.
Mr Beer: If we scroll down we can see what Mr Stewart asked. He cuts in his explanation of the position, yes?
Anne Chambers: He’s pasted in my update from PinICL there. And then, where it says, “I notice”, that, I believe, is his words from that point.
Mr Beer: Yes:
“I notice that in the early guise of this problem in the call it states the PM as female.”
Yes?
Anne Chambers: Yes, that’s what Mike’s saying.
Mr Beer: Then some more cutting and pasting.
Anne Chambers: Yes, then –
Mr Beer: Then back to him –
Anne Chambers: – that’s what the helpdesk had put on the call, yes.
Mr Beer: “At the bottom of this email re a magical [£43,000] appearing and disappearing the PM is Male He reports”, et cetera.
Anne Chambers: Mm-hm.
Mr Beer: Then scroll down, please. He says:
“Clearly the [subpostmaster] is concerned, as we have just spent a number of months trying to sort out the first instance and he doesn’t want a repeat performance. He is convinced that there is something wrong with his Horizon kit. I would be grateful if you could investigate this and give him any support you can. I’m due to visit the office tomorrow to look at his paperwork and discuss the situation …”
Anne Chambers: Again, isn’t that again a cut and paste from something that somebody in the Post Office had said earlier in the chain?
Mr Beer: I’m not sure that it is, I thought that was –
Anne Chambers: I don’t think Mike would have been visiting offices.
Mr Beer: No. I think you’re probably right, then. Can we go to your answer then, please, on page 3 of this email chain. At the foot of the page. You respond:
“Mike …
“Haven’t looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks …”
Is it right that this Riposte problem had been around for years?
Anne Chambers: Yes, because we had been seeing it since at least the end of 2000.
Mr Beer: So five and a half, six years?
Anne Chambers: Yeah.
Mr Beer: Your witness statement – there’s no need to turn it up, I don’t want to disrupt the narrative – says at paragraph 76, “[You] personally had known about this Riposte lock problem since soon after I arrived at the SSC in 2000”, and that’s a reference back to those November 2000 PEAKs we looked at; is that right?
Anne Chambers: Yes, although I think that when I wrote the witness statement, I’m not sure that I’d actually seen the dates on those but, yes, it was a known problem.
Mr Beer: So that reflects the early PEAKs that we saw of November 2000 –
Anne Chambers: I believe so.
Mr Beer: – and an early KEL that we saw of November 2000?
Anne Chambers: Yes.
Mr Beer: In relation to your part of the sentence which says, “This affects a number of sites most weeks”, how did you know that it affects most sites – sorry, it affects number of sites most weeks?
Anne Chambers: Because of the event storms that we could see. I would say that I think there – it was something that did seem to have increased, as time went by. I don’t believe we were seeing all these event storms several times affecting many branches all the way through, although, actually, since I wasn’t necessarily checking, I don’t know. But when say it affects them, I mean that we could see, if we looked, that event storms were happening. It does not mean that it necessarily had any impact on their branch accounts. I’m not saying that every week a number of sites were having incorrect discrepancies because of this problem.
Mr Beer: But you didn’t know one way or the other?
Anne Chambers: We would have known – okay, it’s slightly peripheral. Some aspects, including the transfer problem, and the rolling over without the transactions included, would have caused entries on the reconciliation reports, certainly up to 2005. So that would – those would all have been investigated as they happened. And I’m certainly not aware – I don’t remember that every week we were having a branch or two with those reconciliation report entries –
Mr Beer: But, Mrs Chambers, this is a problem that’s been going on for five or six years.
Anne Chambers: Yes, and if we had been getting all those reconciliation report entries at that sort of level for five years, it would absolutely definitely have been picked up and seen as being a big ongoing problem.
Mr Beer: But isn’t that relying on subpostmasters – to an extent, relying on subpostmasters calling in?
Anne Chambers: Not the reconciliation report entry reports.
Mr Beer: No, the other part of the answer that you gave –
Anne Chambers: The other part of them phoning and saying, “Oh, I’m doing my balancing and it’s all gone haywire”? Yes, for us to know about it, they would have had to phone in and that call would have had to be passed to SSC and it’s quite possible/probable, that the majority of those calls were not passed to SSC because they were just being told to reboot.
Mr Beer: Yes. Did you know how many sites were affected every week?
Anne Chambers: No, I could have known if somebody had asked me to monitor that and, obviously, at the point that I was doing some analysis, then I did know. And I think I found – I can’t remember what period it was, in that previous KEL we looked at – previous PEAK we looked at, where I did do some monitoring and I’d found two branches with the event storms. But, as I say, I can’t now remember what length of time that was over.
Mr Beer: You continue in that sentence:
“… and finally Escher say they have done something about it.”
Anne Chambers: Yes.
Mr Beer: In your witness statement, you say that:
“I and others in the SSC understood the cause of the problem was to be a problem in the Riposte software, which we thought was being investigated by Escher.”
Of the five or six-year period that we’re looking at, for how long had you thought that Escher had been investigating the issue?
Anne Chambers: I thought they’d known about it all the time. I now think – well, I now know, since putting all the calls together, and so on, that that’s extremely unlikely.
Mr Beer: What had gone wrong?
Anne Chambers: Um, it was – nobody was managing it as a problem. It was almost impossible for SSC staff to see which calls were with Escher and who was progressing those, because they sort of went on to a separate PEAK stack, which I now know, yes, we could see but I don’t think at the early days I knew quite where it was. And it wasn’t SSC’s job, really, to be monitoring those, but I’m not sure whose job it was.
Yeah, I think if we’d appreciated that nobody effectively was looking at this for all that time, we would have flagged it up and jumped up and down. But that realisation just didn’t come until late in the day when, finally – you know, we did send a call over. It did get picked up, eventually, and sent to Escher and they did produce a fix.
Mr Beer: You say you were interested in whether they have really fixed it.
Anne Chambers: Oh, I never believe anything anybody tells me.
Mr Beer: Was that –
Anne Chambers: You check; you double check; you triple check.
Mr Beer: Was that more directed to what you knew about Escher, rather than being cynical about the world in general?
Anne Chambers: No, I was cynical about the world in general.
Mr Beer: You therefore left the call open, the PEAK open?
Anne Chambers: Yeah.
Mr Beer: If we turn up paragraph 81 of your witness statement, please, on page 26. Paragraph 81 at the top. You say:
“I am asked whether Post Office or subpostmasters were told about the problem. It was not raised as a wider problem with Post Office; each instance was treated individually.”
Does that mean, until that email chain that we picked up a moment ago, that Post Office was kept in the dark for the best part of six years.
Anne Chambers: I’m not sure they were intentionally kept in the dark but I think they were in the dark, yes.
Mr Beer: You say each instance was treated individually. Why was each case treated individually?
Anne Chambers: Because we would look at the calls that did come through, where they came through to us, and if there was an effect on the branch accounts, then we would pass the information via MSU to Post Office on a BIMS report or it was passed that way.
Mr Beer: This had been a problem that had been around for five or six years. Would you accept that Fujitsu had failed properly to investigate and address the bug?
Anne Chambers: Yes.
Mr Beer: And failed to tell – I’m not saying that you personally failed to tell – the Post Office about its existence?
Anne Chambers: Yes, I think because it was always treated as individual instances, it wasn’t raised as a problem and flagged through to Post Office.
Mr Beer: You said in your email that you wanted to wait and see to see whether the S90 release was an effective fix. Was that a case of waiting to see whether any more calls came in from subpostmasters and, if so, whether any of those calls could be linked to the Riposte lock problem?
Anne Chambers: No, I monitored the events coming through the system to see if there were any more of these event storms occurring anywhere.
Mr Beer: To complete the loop, if we may look at a last couple of documents. FUJ00083667. This is a PEAK, 0127246, opened on 12 October 2005. If we look at page 3 and the entry for 11.14.22 at the top of the page. You say:
“Since [the release] S90 was distributed, the number of these timeout events over the whole estate has gone right down, with no storms from an individual counter. So it looks as if the Riposte change has been effective.”
Anne Chambers: Yes.
Mr Beer: Was that your measure of working out whether the fix had been effective?
Anne Chambers: Yes.
Mr Beer: Was there any other means of working out whether the fix had been effective?
Anne Chambers: I think that was a reasonable check of whether the fix had done what was expected of it. Obviously if, after this, we got more calls about balancing issues, failure to replicate across counters, and so on, then they would be investigated from scratch again.
Mr Beer: Lastly, can we look at an email exchange, please, between you and Mr Jenkins from 2010. That’s FUJ00083722.
If we look at the email at the foot of the page, please. Can you see your email, from you to Mr Jenkins?
Anne Chambers: Yes.
Mr Beer: You forward, I think, if you look underneath it, the chain from back in 2006 that we see?
Anne Chambers: Yes.
Mr Beer: Why were you doing this?
Anne Chambers: I presume he had contacted me and asked me what I could remember about the Callendar Square problem.
Mr Beer: Can you remember why, in February 2010, Mr Jenkins would be contacting you about what had happened to the Callendar Square bug back in 2006?
Anne Chambers: I – no, I don’t remember specifically why. It might have been because he was involved in a prosecution and wanted to know some of the details in case this was raised. I did my best to provide some information for him.
Mr Beer: So let’s see what you told him, please. If we just scroll up, please. You say, “Gareth”, and you give a reference to a KEL:
“I’d forgotten – this did give a discrepancy, but also a receipts and payments mismatch, if they persisted and rolled over (though it was usually obvious that something was wrong).
“And a flood of NT events (not ‘Riposte events’!) which SMC should have noticed at the time.
“Since we are now checking for these particular events, and did a catch-up for old retrievals, can you say that the current branch did not have a problem??”
What were you referring to when you said “the current branch” –
Anne Chambers: That suggests he was asking me about some specific branch and could we see if they had had the Callendar Square problem or not. But I now don’t know which the “current branch” would have been.
Mr Beer: Would you interpret that as meaning “Can you say, in the evidence that you are to give, that the current branch did not have this problem?”
Anne Chambers: I cannot remember precisely why he was asking me but that is a possibility.
Mr Beer: You say:
“Anyway it stopped happening once S90 was installed …
“This particular problem would only affect branches with more than one stock unit. It happened several times at Callendar Square, though we never found why they were so badly affect.
“Is this sufficient?”
The line, “This particular problem would only affect branches with more than one stock unit”; was that accurate?
Anne Chambers: That was accurate if we were talking about the Callendar Square part of the Riposte lock problem. Callendar Square had had the very specific problem of being able to do the transfer ins multiple times, so that’s what I was referring to there. You could only do transfers out and in if you had more than one stock unit. There may well have been a separate email or two or discussion behind this specific email but I cannot remember now.
Mr Beer: Then scrolling up to see what Mr Jenkins’ reply was:
“Thanks Anne,
“Penny …”
Is that Ms Thomas?
Anne Chambers: I would think so, yes.
Mr Beer: “… pointed out on Friday that [the Post Office] have not asked us to retrieve any data for this branch yet! Therefore we have no message stores to check against Event Logs.
“This will probably do me more now.”
Does the case of Seema Misra ring any bells with you?
Anne Chambers: I’ve obviously heard the name, yes.
Mr Beer: Is that recently seen the name or something you recall being asked about back in 2010 and Mr Jenkins’ preparation to give evidence in the Seema Misra case?
Anne Chambers: I think it’s likely I had some involvement back then but I cannot remember definitely which case that was.
Mr Beer: Can you remember the context of this exchange, ie he was asking you for what he should say or things to say by reference to your work and your understanding of the issue in the Seema Misra case?
Anne Chambers: I thought he was just, sort of, checking with me just to see what I knew in general about the problem, and so on, what my recollection of it was.
Mr Beer: Thank you. Mrs Chambers, they’re the only questions I ask you at the moment. I’m going to have to draw a line under Bug 2 and ask you about the remainder of things when you come back in the future for your Phase 4 evidence. I need to allow reasonable amount of time for other people to ask their questions. Thank you very much for the evidence you’ve given.
Sir, might that be an appropriate moment to take a ten-minute break for the transcriber – 15-minute break, she’s mouthing to me – until 3.35?
Sir Wyn Williams: Well, on the basis of the guesstimates that you were provided by your colleagues yesterday, that should provide sufficient time to finish by around about 4.15?
Mr Beer: Yes, I think so.
Sir Wyn Williams: All right.
Mr Beer: Thank you very much.
(3.20 pm)
(A short break)
(3.34 pm)
Mr Beer: Sir, thank you. Can you see and hear us?
Sir Wyn Williams: Yes, I can.
Mr Beer: Thank you very much. I think Mr Stein is going to ask questions first.
Sir Wyn Williams: Right.
Questioned by Mr Stein
Mr Stein: Mrs Chambers, I ask questions on behalf of a large number of subpostmasters and mistresses on behalf of a firm of solicitors called Howe+Co.
You’ve given evidence and been asked a large number of questions by Mr Beer, so I don’t need to cover those areas.
You said, as part of your evidence that on some KELs you would see associated issues but not all. You also said, as part of your evidence today, that sometimes you would expect to your colleagues and then you would discover that they’d been dealing with another matter in the last few weeks. Now, help us a little bit with the question of duplicates. There was a system that we know that the first tier of the helpline were discouraged from sending through duplicates to the third, fourth lines. Now, would you see on your KELs that there were duplicates?
Anne Chambers: You mean if the Helpdesk had used the same KEL for various tickets that they had received?
Mr Stein: If they had recognised what they thought or believed was a KEL that was the same as one that was already being dealt with, would you see that?
Anne Chambers: Not automatically, no. But, obviously, some KELs would say, “Send the call to SSC”. Even though it’s logged as a known problem, we would still make it clear on the KEL that we still needed to see the call, if that was thought to be appropriate.
Mr Stein: So if you didn’t see some of the issues that were coming through, because they were thought to be duplicates and therefore shouldn’t go to third and fourth line support, does that mean that you were not always aware of how many calls were being made through to the helpline on the same issues?
Anne Chambers: That’s true, if it was something that there was a resolution or possibly a workaround, which they could tell the postmaster about themselves.
Mr Stein: Who provided the training to the first line support, the first line Helpdesk call answerers so they could recognise that this was the same or a different KEL; who provided that training?
Anne Chambers: I don’t know who provided their training.
Mr Stein: Were you ever asked to provide such training?
Anne Chambers: No.
Mr Stein: During the evidence in this Inquiry, many of our clients, who are ex-subpostmasters and mistresses said that their accounts, branch accounts, never seemed to balance. So that was Janice Adams; Mujahid Faisal Aziz explained that there were very any shortfalls that they had to balance by paying in cash; Edward Brown said similar matters occurred to him and that it wasn’t always a large shortfall but sometimes it could be into the thousands; and Gary Brown reported that the shortfalls happened so often that it was hard to keep track.
Can you help us understand how it was that the subpostmasters and mistresses experienced so many shortfalls?
Anne Chambers: No, I can’t, and it obviously concerns me if this was happening and they weren’t being given assistance by anybody to get to the bottom of these problems.
Mr Stein: You’ve mentioned at the closing part of your statement, paragraph 212, I’ll read it out, that:
“A point of frustration with the system was that the users, namely the subpostmasters, were not our clients, and there was a practical limit as to the extent to which we could work together with them to investigate problems.”
Was that true, this difficulty, having that separation?
Anne Chambers: Yes. I mean, we had no ability to find out what was actually taking place at the branch. I’m not necessarily saying it was their – they were making errors that were causing these problems but where we were checking, by one means or another, that the system was correctly calculating discrepancies based on the transactions that had been recorded on the system for the branch, then if that calculation and those checks show no problems, then you’ve got to try to find out what has been recorded in the branch accounts or what is missing in the branch transaction list which is causing the discrepancy.
And unless you’ve got some way of going to a branch and actually finding out what should have gone onto the system, then you cannot identify the cause of the discrepancy.
Mr Stein: When you say at paragraph 212 that the subpostmasters were not your clients, your client was the Post Office?
Anne Chambers: Yes.
Mr Stein: Excuse me one moment.
Thank you, Mrs Chambers.
Questioned by Mr Moloney
Mr Moloney: Mrs Chambers, one matter and it’s just one document, FUJ00138385.
Anne Chambers: I’m sorry but I can’t hear you very clearly.
Mr Moloney: I’ll speak up and get closer to the microphone. Sorry, Mrs Chambers.
Anne Chambers: Thank you.
Mr Moloney: It’s just one document, FUJ00138385, and it should go onto the screen – and it’s on screen now. Thank you.
The subject line, the title, is “Requesting journal data from Audit”, the author is you, Mrs Chambers, and it’s created on 25 August 2011.
Anne Chambers: Yes.
Mr Moloney: So that’s after the migration to HNG-X or Horizon Online –
Anne Chambers: Yes.
Mr Moloney: – however one terms it. We see from this document that:
“All journal messages arriving at the data centre are retained for audit. Occasionally SSC may need to ask for data to be retrieved to enable issues which happened more than six months ago to be investigated.”
Then in brackets:
“(Less than six months, there may still be sufficient and more accessible information in BRSS).
“When asking for transactions for a FAD …”
That’s a branch, isn’t it?
Anne Chambers: That’s a branch.
Mr Moloney: “… and date range, ask for the QUERY_ATFINALFilteredhx.xml file which will contain all transactions for a given date range and FAD code in XML format. These will however lack the JSN and ReqMessageID.
“The alternative is raw files containing data for all 80 or so branches with the FAD hash, which is far harder to read but does contain JSN and ReqMessageID.
“Route the PEAK with the request to Security Ops.
“If you think the call may be part of a Post Office investigation into a branch that might lead to litigation, then this should not be handled by SSC unless already authorised by the SSC manager.”
So it follows from this document, as we are already aware, that different types of audit data provided different information to analysts?
Anne Chambers: Yes.
Mr Moloney: So here, it refers to raw files of data or raw data?
Anne Chambers: Err …
Mr Moloney: As one type of data available?
Anne Chambers: One type of data is the raw files, yes.
Mr Moloney: And then there’s the XML files, as well?
Anne Chambers: Yes.
Mr Moloney: The raw files of data, the raw data, would contain information not contained in the XML file?
Anne Chambers: It – yes, I cannot now remember precisely what the details are but, obviously, there’s two fields there that might have been of use.
Mr Moloney: Yes. How frequently would raw data be requested in your experience?
Anne Chambers: I can’t remember ever personally actually requesting journal data from Audit. I almost certainly did on occasion but I’ve got no memory of doing it. It certainly wasn’t a frequent thing.
Mr Moloney: Why would the PEAK and request have to go to Security Ops, as is said in the third line from the bottom?
Anne Chambers: Because they were the ones who could access this data and they had to extract it from the audit servers. Only they could do that.
Mr Moloney: Right. How would an SSC analyst be aware that a request might lead to litigation?
Anne Chambers: Not might lead to but might already be part of a Post Office – sorry, yes. Um, it would just depend, you know, where the call had come from, if anybody had mentioned this. I don’t know. I mean, how we would tell that now, I’m not 100 per cent sure.
Mr Moloney: Yeah. Because this does say that:
“If you think the call may be part of a Post Office investigation into a branch that might lead to litigation, then this should not be handled by SSC unless already authorised by the SSC manager.”
Why would the request not be handled by SSC unless already authorised by the SSC manager?
Anne Chambers: I presume I was told that. I’m not sure. I mean, this is an unusual situation if we are asked to investigate something that happened more than six months ago. Normally, we’re investigating – would have been investigating things that had happened recently. So I presume that was what I was told, and that is why I added that into the work instruction. But I cannot remember of any conversation about that.
Mr Moloney: So this is a work instruction, is the term you’ve just used; is that right?
Anne Chambers: Sorry yes, a “WI” is a work instruction.
Mr Moloney: Work instruction. You would have been told that if it was part of a Post Office investigation into a branch that might lead to litigation, then it shouldn’t be handled by SSC, unless already authorised by the SSC manager?
Anne Chambers: I appear to have included that in the work instruction, so I assume that was what I was told.
Mr Moloney: So did you understand why it was that you were told that this should be included in the work instruction?
Anne Chambers: Because presumably, in that case, Post Office would be putting in their own request for the data.
Mr Moloney: Why would that need to be authorised by the SSC manager?
Anne Chambers: Just because I was told that. I’m sorry, I have no real recollection of this. I don’t recall being told it. I don’t actually recall writing the work instruction but I obviously did.
Mr Moloney: Just to float one possible reason, could it be a reason of payment for this type of request?
Anne Chambers: That may have come into it because I think Post Office were – they were charged for data retrievals but, yeah, I think this would be better directed at my – one of my managers, probably.
Mr Moloney: Of course. I only ask you because you’re the author of this document, Mrs Chambers.
Anne Chambers: Yes. I expect Steve or somebody said, “Oh, this ought to be in a work instruction, can you create one?”
Mr Moloney: That was the final question I had for you, which is: in what capacity were you providing direction of this nature to your colleagues?
Anne Chambers: Obviously, I would have a good understanding of the technical messages and the content of the messages, and so on, or, you know, a reasonable understanding of that. And, yeah, I’m – I wrote a work instruction. I’m sorry, I don’t really remember any more about it than that.
Mr Moloney: Given that you didn’t know really why that final three lines, or rather final two lines, were included within this work instruction or you can’t remember why, would this have been better being a work instruction emanating from your manager rather than from you?
Anne Chambers: Um, yes, it probably would have been, but if a work instruction was felt to be needed for something, then somebody might well be asked to just “Oh, could you write that work instruction?” But that didn’t mean that all the content necessarily came from me. We could all write work instructions and it was the sort of job that got shared out amongst us.
Mr Moloney: Thank you very much, Mrs Chambers.
Questioned by Ms Page
Ms Page: I’m so sorry, can I just ask those in front of me to just sit as they normally do, so I can see the witness. Thank you very much.
Anne Chambers: I still can’t hear you terribly clearly.
Ms Page: Is that any better?
Anne Chambers: That’s better, thank you.
Ms Page: It’s Flora Page, also acting for a number of subpostmaster Core Participants. I’d like to take you to two documents. They’re both PEAKs or possibly one might be a PinICL and they both deal with the process that you went through in order to insert transactions to ensure that the accounts were balancing properly.
So the first one is FUJ00152239. What we can see is that the summary at the top shows us that the office can’t balance as there are “incorrect fees on POs”, and I think that stands for Postal Orders?
Anne Chambers: Yes, it does.
Ms Page: We can see that this dates from July 2001. If we then – I don’t think we need to see anything in the first few dealings but if we go down to page 4 and we can see that it starts off in your department with Barbara Longley, who’s the administrator; that’s right, isn’t it?
Anne Chambers: Yes.
Ms Page: Then we see that John Simpkins initially picks this up, and we can see that he has – or somehow, about halfway down the page, he has noticed – he says:
“PRESCAN: Check date/time runs in message store for time BU was swapped.”
So we can see a base unit has been swapped out; is that fair?
Anne Chambers: Yes, that would be fair.
Ms Page: Then if we go a bit further down, John Simpkins has assigned this to you, “Team Member: Anne Chambers”, or somehow it has been assigned to you; do you see that?
Anne Chambers: Yes, I do.
Ms Page: Then he says that it might be:
“… a problem due to the corrupt storage unit, check the message store for any corrupt entries then insert a REM OUT for PO Fees …”
Anne Chambers: I see that.
Ms Page: Yes, but if we then go over the page, you seem to take a slightly different view at one point, but we’ll go through it logically. At 15.41, you say:
“It looks as if Adjusts Stock on 4th Jul was showing incorrect figures …”
Then you’ve referenced a KEL.
“As a result, the PM did couple of sets of unnecessary SAPs …”
After base units swapped, it seems.
Then if we go down almost to the bottom of the – I’m so sorry, to about four lines below that, you say:
“I’ve raised OCR AChambers … to allow us to correct the messagestore.”
So that’s an instance, is it, of you saying that you need to go through the change control process to insert transactions; is that right?
Anne Chambers: That appears to be what I did. I have to say I have no recollection of this at all.
Ms Page: I wouldn’t be surprised, it’s obviously going back a very long way, isn’t it?
Anne Chambers: Yeah.
Ms Page: Then it says, below that:
“Incident Under Investigation.”
Then, if we go further down, we can see “New evidence added”, and I’d just like to try to understand what these evidence types are.
We’ve got:
“New evidence added – Full message store.”
Then we’ve got:
“New evidence added – audit logs.”
Then we’ve got:
“New evidence added – PSStandard logs.”
Is “full message store” the equivalent of what became the ARQ data?
Anne Chambers: No, it’s not. It would include all the ARQ data but it’s the – all the messages for the branch that were in existence on the day and time that I did the retrieval. I’d have retrieved it from the copy of the message store that was held on the correspondence server centrally and so it’s all the transaction messages and a lot of other messages that have been written in the last 42 days, it would have been at this point, plus all the reference data relating to the branch. But the ARQ data is the same data but it’s captured in a different way.
Ms Page: That last bit you said is the bit I wanted to get at. It is the same data, is it?
Anne Chambers: It’s the same data which I’d retrieved from the correspondence server but the messages, as they came in from – I think it was happening all the time – as they came in from the branch to be fed into the correspondence server message store, they were also – there was a stream of them going out all the time into the audit files. So the ordering, in particular reference data, and so on, would be rather different but, overall, it’s the same data.
Ms Page: So that’s the evidence that could be captured for a significant period of time afterwards and that was stored –
Anne Chambers: The ARQ data files were kept for a significant amount of time. The message store, some of the messages persisted, but others would be archived or deleted after 42 days.
Ms Page: All right, so it’s not identical. All right. Well, then, audit logs. What is that? Is that identical with ARQ or not?
Anne Chambers: No, this is the files that I had totally forgotten about until Mr Beer reminded me of their existence yesterday, I think it was, which were kept on the counters, were written on the counters, and contained a certain amount of diagnostic information written to the file by the counter application as things were done. I can’t – I’ve got very little recollection of precisely what that looked like.
Ms Page: Then PS standard logs, what are they?
Anne Chambers: That’s another counter log file in which you could see messages to and from the counter peripherals. Things like the printer and the barcode reader, and so on.
Ms Page: None of that was kept for any significant period of time; is that fair?
Anne Chambers: No, that wasn’t kept and it normally wasn’t retrieved from the counter. It was only if we were investigating something we would get the file from the counter.
Ms Page: Yes. All right. Then further down, we can see that you’re asking for Development to look at this and then, if we go over the page, you say you haven’t been able to reproduce it:
“This counter had a box swap an hour before the problem occurred but I don’t think that is relevant.”
You’ve then said that you’ve got another report of the same problem elsewhere and you give the forward number for the branch:
“… so please can this be looked at quickly (especially as if it is not reported before rollover, we have to get POCL authorisation for the fix and so it is very visible).”
That means, does it, that if you had to fix it after rollover by inserting transactions, then it would be visible in some way?
Anne Chambers: I’m struggling to remember any details of this. I think if it was reported, but not until after the rollover, we probably possibly couldn’t have fixed it at all. There would have been a receipts and payments mismatch, and then we would just have reported it to Post Office through the MSU BIMS route. But I can’t be certain about this, having no memory of it, and I don’t think I’ve seen this document until this moment.
Ms Page: The word there, “visible”, visible to whom, do you think?
Anne Chambers: I don’t know why I used that word. Yeah, visible to Post Office, I suppose. But, as I say, I have no memory of this.
Ms Page: Or visible to the subpostmaster?
Anne Chambers: Um, I think the subpostmaster knew about the problem already because they had reported the problem.
Ms Page: True, but you’ve identified that there is another problem elsewhere.
Anne Chambers: Yes. That had been reported to us as well, so that was another postmaster who had noticed that it had happened. I mean, the summary is that they can’t balance. I can’t remember if it actually totally stopped the process because of this inconsistency, or if they could push on. I mean, obviously Postal Orders and the fees associated with them should always be in step with each other. You shouldn’t be able to have one without the other, and something had gone wrong here, and they were out of step.
Ms Page: A little further down, it says that there may be – the counter is M1 and M1R. You may not be able to recall what that means. But you do say – sorry, Les Ong says:
“There are two fixes that I know of … relating to Postal Orders that could have a bearing on this …”
Then, if we go over the page, what we see when it comes back to you – and this is on 10 July, so it’s a subsequent day – at 15.31:
“Authorisation for messagestore amendment now received from …”
Then we seem to get an email address, “mick.theobald”, and it has been edited out so we don’t have the full email address. Is that a name that rings a bell?
Anne Chambers: I think he was a Post Office person but I can’t be a 100 per cent sure.
Ms Page: Following that and a little further down, we see that:
“Applied fix to message store …”
Then there’s the reference to the OCR again, and:
“Balance snapshot now shows 19 POs and fees. Leaving call open until balancing/cash account done.”
So it looks as if it has been possible to apply this before rollover.
I suppose the question that I’d like to ask is how would it be possible to see, on the ARQ data, what had happened here?
Anne Chambers: Right. In the ARQ data, you would see, when the initial problems were happening, the SAP – stock adjust positive and stock adjust negative – lines that related to where they were trying to adjust their stock of Postal Orders which they’d documented, and it was in there that the amounts got out of step in that one, for the Postal Order itself it was for a certain quantity, and for the fee it was for a different quantity. So that’s where the problem arose.
And then, in the ARQ data for 11 July, you would see the transaction also affecting the postal – presumably affecting the Postal Order fees product, which I’m guessing now, but I imagine was another stock amount – stock adjust transaction for the difference that was wrong. Whether there is anything on that individual message which, in the ARQ data, enables you to know that it was me who did it and not somebody at the branch, I do not now know, because I have got no record in this PEAK here of exactly what it was that I inserted.
It’s possible, as I said before, sometimes we used a dummy counter number. Sometimes we inserted a comment, but that is not necessarily going to be visible in the ARQ data as retrieved. Sometimes we used a username to try and make it obvious that it was SSC who had made the change but it’s not recorded on the PEAK here precisely what was done. Those messages would have been captured somewhere and recorded for posterity but I don’t know where.
Ms Page: The transactions would be asynchronous, would they, in the sense that the balancing transactions that you’ve inserted would show the date that you inserted them, not backdated?
Anne Chambers: Yes.
Ms Page: All right. Thank you. If I may just then briefly – the next one is a bit quicker. If we look at FUJ00152240. We can see this is summarised as “Cannot put transfer through”, and this dates from 2004. The last entry that we can currently see says:
“PM reports that he cannot put a transaction through it keeps coming up with an error message.”
If we then pick this up on page 2 at 12.30, if we scroll down a little bit, this is where Barbara Longley has assigned it to you and you’ve stated that:
“The transfer causing the problem was started while the user was attached to the SU BDC.”
Can you just remind us: SU BDC?
Anne Chambers: Stock unit called BDC, which was most likely Bureau de Change.
Ms Page: That certainly seems to be an issue with this one. There does seem to be problems with it being foreign transfer. Then you’ve said, in the last paragraph of this entry:
“I’ve spoken to the PM and asked him not to balance stock units BDC, MM or MC until we have sorted out the problem. I’m loading up the messagestore on a test counter and hope that by amending the EPOSSTransfers object it will then be possible to reverse the transfer.”
If we scroll down, just going over the page line:
“I’ve made a messagestore correction …”
So you’ve then given an OCR reference so it looks as if, again, we’ve got this process where you’re formally seeking the approval to insert a transaction; is that right?
Anne Chambers: That’s what it looks like. Again, I’ve not seen this and I’ve got no recollection of it, but it looks like it.
Ms Page: It says:
“Before and after messages attached.”
Is that a practice that rings a bell?
Anne Chambers: Well, we always – yes, we’d always make a record of what we were changing or adding in. I don’t know precisely now what they look like.
Ms Page: When you say “attached”, is that attached to the PEAK?
Anne Chambers: Yeah.
Ms Page: So the PEAK would have had the messages before and after the message you inserted attached to it; is that right?
Anne Chambers: That’s what it sounds like, yes.
Ms Page: You’ve then recorded:
“Have spoken to PM and informed him he should be able to continue with the balance now.”
Anne Chambers: Yes.
Ms Page: What you don’t say is “I’ve informed him I’ve inserted a transaction into your account”, do you?
Anne Chambers: I don’t explicitly say that, but I imagine I would have explained to him what I had done, and that I had removed the transfer, which I – I mean there’s – it’s – I’m not clear from this whether I wrote a pair of corrective messages that actually removed – how would it have done it? Um, I’m not clear. Further up in the call there was mention of an EPOSS transfers object and suggesting that that needed to be rewritten in order to let this progress. So I don’t know if that’s what I actually did, or whether I did insert a pair of opposite messages to undo the transfer that was outstanding.
I certainly wouldn’t have hidden from him the fact that I was changing something on his system which would remove this transfer that was preventing him from balancing his office and continuing to trade, to do his normal business.
Ms Page: Not hidden, but maybe not mentioned in the sense that it’s not recorded?
Anne Chambers: It’s not written down but that doesn’t mean I would have said it because I usually did. I didn’t make a secret of the fact that system problems happened, and I think it was perfectly – I’m sure it was perfectly clear to him that there had been a system problem to do with a transfer which I then – and then, once I had done whatever it was I did, the transfer that he didn’t – that was stopping him had been removed in some way in order for him to progress.
The fact I didn’t write it down does not mean I said absolutely nothing to him. I certainly wouldn’t have phoned him back and said, “Oh look, it’s miraculously all okay now, you don’t need to bother any more”. I wouldn’t have approached it in those terms, but I do not know precisely what I said to him.
Ms Page: Well, a system problem is one thing, but actually inserting transactions into the data that’s stored on his counter is a different thing, isn’t it? If it’s not written and recorded here, how would he – how would posterity ever know that you’d ever said that to him and told him that’s what you were doing?
Anne Chambers: If I’d known posterity was going to be asking I would have written it down. But I don’t know if there’s any more information in the OCR. That certainly would make it clear exactly what it was that I did.
Whether I explicitly said, “I have removed the transfer, I have accessed your counter transactions and removed the one that was causing the problem”, which is effectively what I’d done, I’m not sure it even needed saying. I would have thought he would have realised that that was what I had done. But I cannot – I do not know what I said.
Ms Page: Thank you. Those are my questions.
Sir Wyn Williams: Is that it? Anyone else have any questions?
Mr Beer: No, they don’t, sir. Thank you very much.
Sir Wyn Williams: Right. Well, thank you, Mrs Chambers, for giving detailed answers to detailed questions over two days. As you know, you will be asked to return at some future date. I don’t think we can yet tell you what that date is. If you haven’t already received it, the probability is that you will get another Rule 9 Request so that the general questions will be provided to you in advance and although, in a sense, you’re in the middle of giving your evidence, it’s unreasonable for me to expect that you don’t have access to your lawyers if you want to have access to your lawyers.
So unless anybody immediately shouts out and says to me “You can’t do that”, I’m now going to tell you that if you want to speak to your lawyers, you can. All right?
Anne Chambers: Thank you.
Mr Beer: Thank you very much, sir.
For reasons that you know, we return on Tuesday next week, 9 May, to hear evidence from Barbara Longley at 10.00 am.
Sir Wyn Williams: All right. Then the Inquiry is adjourned until then. Thank you all very much.
Mr Beer: Thank you, sir.
(4.12 pm)
(The hearing adjourned until Tuesday, 9 May 2023 at 10.00 am)