2 May 2023 – Anne Chambers
Hide video Show video
(10.00 am)
Housekeeping
Sir Wyn Williams: Mr Beer.
Mr Beer: Sir, before we hear the evidence of Mrs Anne Chambers, I should inform you of a significant development in relation to the evidence of Gareth Jenkins, from whom we were due to hear evidence on Thursday and Friday of this week. As you know, the Inquiry served a written request pursuant to Rule 9 of the Inquiry Rules 2006 on Mr Jenkins seeking a witness statement from him in relation to the issues that arise in Phase 3 of this Inquiry on 31 August 2022.
That request contained 56 numbered questions, although many of the questions contained sub-questions, meaning that some 200 questions were in fact asked, set out over 17-odd pages. After resolution of the issue of whether you should write to the Attorney General to ask her to provide an undertaking that any evidence given by or produced to the Inquiry by a person would not be used against that person in criminal proceedings against them was resolved, you determined that you would not presently seek such an undertaking but would keep the position under review.
Mr Jenkins provided a witness statement to the Inquiry on 6 February 2023. That witness statement was short, 15 pages in length, and addressed a small number of the questions that had been asked of Mr Jenkins. It did not address the issue of Mr Jenkins’s knowledge of, and involvement in, the investigation of a series of bugs, errors and defects in the Horizon System. That issue was the principal issue raised by the Inquiry’s written request.
In particular, it didn’t address at all questions 16 to 49 in the Inquiry’s written request, all of which went to the issue of knowledge of bugs, errors and defects.
In relation to the questions which were not addressed in the witness statement, Mr Jenkins said:
“I wish to maintain my reliance on the privilege against self-incrimination in relation to certain of these questions for the time being but I want to make clear my reasons for doing so. As matters stand, my lawyers are working their way through the evidence disclosed to Core Participants by the Inquiry and will continue to do so. I understand that, at the present time, they are unable to advise me fully on the privilege against self-incrimination, particularly in relation to Phase 4, because their review is ongoing and because the existing evidence disclosed in relation to Phase 4 is limited. I wish, therefore, to ensure that it is understood that I will continue to keep the question of whether I waive my rights under consideration.
“My lawyers have undertaken to the Inquiry to notify it in good time when they are in a position to advise me fully. I will continue to prepare for the Inquiry so as to ensure that when my lawyers are in a position to advise me, that is not a cause for delay.”
On Friday of last week, the Inquiry received a letter from Mr Jenkins’ solicitors stating that they and Mr Jenkins had continued to review the material disclosed by the Inquiry, that although disclosure was a rolling process, and disclosure in relation to Phase 4 in particular remained very much ongoing, they had a greater understanding of the evidence that relates to Mr Jenkins in Phase 3 and that, having had the opportunity to consider the disclosure, Mr Jenkins wished to indicate his willingness to provide the Inquiry with a response to the request that was served on him on 31 August 2022 in a statement which addresses all of the questions asked in that request.
This is plainly an important development. In the light of it, you decided that Mr Jenkins should not give evidence on Thursday and Friday of this week for three main reasons: first, that it was in the interests of the Inquiry, of the Core Participants and of the public that the Inquiry should receive the widest range of evidence possible, in particular from a central figure such as Mr Jenkins. It was therefore important to obtain this written evidence from Mr Jenkins.
Secondly, it would have been unfair on Core Participants, and in particular the subpostmaster Core Participants, for Mr Jenkins to have entered the witness box on Thursday and Friday of this week without him having previously provided a written account of what he proposed to say about errors, bugs and defects in the Horizon System.
It’s important that all Core Participants have the opportunity properly to prepare for witnesses and giving evidence without previously having made a statement on the issues of substance would have undermined that preparation.
Thirdly, if Mr Jenkins had been permitted to give evidence without having made a statement it would have involved treating Mr Jenkins differently from other witnesses because it may have allowed Mr Jenkins free rein in his oral evidence to say what he wished without having previously reduced his account to writing.
In the light of the time that it’s needed to produce that statement, disclose it to the Inquiry, the Inquiry to assess whether it properly engages with the questions that we’ve asked and to disclose that statement in good time before Mr Jenkins gives his oral evidence, it will not be possible for him to give his evidence in the Phase 3 oral evidence sessions that remain in this part of the Inquiry, ie in the next three weeks.
We will instead interpose Mr Jenkins’ Phase 3 evidence at a convenient moment in Phase 4 of the Inquiry but before the summer break.
Fair notice of this evidence will be given on the Inquiry’s website in the usual way. Mr Jenkins will, in any event, be required to make a written statement about the issues which arise in Phase 4 of the Inquiry, in particular his role in criminal and civil proceedings taken against subpostmasters, and return for a second time to give oral evidence about such matters in Phase 4 of the Inquiry, most likely after the summer break.
It follows that the Inquiry will not be sitting this Thursday and Friday, but the remainder of Phase 3 will continue in accordance with the timetable next week.
Sir Wyn Williams: Mr Beer, thank you very much for explaining to everyone the decision which I took last Friday and the reasons for it.
Mr Beer: Anne Chambers, please.
Anne Chambers
ANNE OLIVIA CHAMBERS (affirmed).
Questioned by Mr Beer
Mr Beer: Good morning, Ms Chambers, my name is Jason Beer and I ask questions on behalf of the Inquiry. Can you give the Inquiry your full name, please.
Anne Chambers: Anne Olivia Chambers.
Mr Beer: Before I ask you any further questions, the Chairman has a statement that he wishes to make?
Sir Wyn Williams: Mrs Chambers, you will have already heard Mr Beer explain that Mr Gareth Jenkins has, in the past at least, relied upon what we call the principle of privilege against self-incrimination.
Before we go any further, I should tell you that a witness at a public inquiry has a right to decline to answer questions if there is a risk that the answer to the question would incriminate the witness.
In short, that is the principle of privilege against self-incrimination.
I remind you of that principle before you give your evidence. I must tell you that it is for you to make clear to me if you wish to rely upon the privilege. If, therefore, questions are put to you by Mr Beer or any other counsel or by me, which you do not wish to answer on the grounds that to answer the question might incriminate you, you must tell me immediately.
At that point, I will consider your objection and determine whether or not to uphold it. I understand that you are represented by a barrister and solicitors today. No doubt if the issue relating to self-incrimination arises, they will assist you.
If, at any stage of the questioning, you wish to speak to your lawyers about that principle, you must tell me immediately, and I will facilitate that.
Do you understand all that, Mrs Chambers?
Anne Chambers: Yes, I do.
Sir Wyn Williams: Thank you very much.
Mr Beer: Thank you very much, Mrs Chambers, for coming to give evidence to the Inquiry today and thank you for providing a long and detailed witness statement to the Inquiry to assist us in our work. We’re very grateful to you.
You should have in front of you a hard copy of that witness statement. It’s in tab A1 of that bundle. It’s in your name and it’s dated 15 November 2022. Do you have that witness statement?
Anne Chambers: Yes, I have.
Mr Beer: If you turn to page 63, please, is that your signature?
Anne Chambers: Yes, it is.
Mr Beer: Are the contents of that witness statement true to the best of your knowledge and belief?
Anne Chambers: Yes.
Mr Beer: For the purposes of the transcript, the URN is WITN01700100. [MS: The actual URN should be WITN00170100.] There’s no need to display that.
Mrs Chambers, I’m going to ask you some questions today and tomorrow about the issues that arise in Phase 3 of the Inquiry. We’re going to ask you to return at a later stage in the Inquiry to answer questions that arise in Phase 4 of the Inquiry, in particular about the role that you undertook investigations of subpostmasters and giving evidence in proceedings bought against them and, in still further particular, the evidence that you gave in the Lee Castleton case. Do you understand?
Anne Chambers: Yes.
Mr Beer: All Core Participants should respect that divide or division if and when they confirm to ask you questions tomorrow and I and the Chair will be keeping a watchful eye to ensure that that process is respected.
Anne Chambers: Yeah.
Mr Beer: I should make it clear that I know that you have spent many hours preparing to give evidence today and have been diligently looking at some of the material that the Inquiry has sent you. You were, I think, provided with a considerable volume of material at the time you were asked to prepare a witness statement on 6 October 2022. You’ve been provided with much more material since then, including in the last two weeks, and I think we’re dealing with such a large volume of material that you couldn’t have hoped to have read all of it and digested it; is that right?
Anne Chambers: That’s right, yes.
Mr Beer: If at any stage I show you a document with which you’re not familiar that hasn’t been part of your preparation, then just say so.
Anne Chambers: Yes.
Mr Beer: Can I start, please, with your career, qualifications and experience. Do you have any professional qualifications that are relevant to the issues that we’re going to discuss in your evidence?
Anne Chambers: I have a degree in statistics and mathematics, which I think shows that I’ve got a reasonable sense of numeracy, and so on, which I think is relevant.
Mr Beer: Probably more than that, a degree in statistics with pure maths, I think –
Anne Chambers: Yes.
Mr Beer: – obtained from the University College of Wales in 1978; is that right?
Anne Chambers: That’s correct.
Mr Beer: Your first employment, I think after graduation, was with a company called Dataskil; is that right?
Anne Chambers: Yes, it was part of ICL. It was ICL’s software house.
Mr Beer: Before you joined Dataskil, did you have any formal qualifications in computing?
Anne Chambers: No, I’d done a couple of computing modules as part of my degree but I hadn’t done a great deal of computing. Like most people at that time, I learnt on the job.
Mr Beer: You say in your statement that at Dataskil you coded and supported various software packages; is that right?
Anne Chambers: That’s correct, yes.
Mr Beer: Was that software concerning databases and statistical processes?
Anne Chambers: Yes, it was.
Mr Beer: To what extent is that the same or different from what you went on to do at ICL and then Fujitsu?
Anne Chambers: Um, it was quite similar in a lot of ways. I mean, I didn’t actually leave Dataskil, as such, it just merged into – it was just subsumed into ICL. So I then carried on working on the same type of things. As the years went by, I did less coding, and so on, and I found I enjoyed the support work.
Mr Beer: You tell us in your statement that from 1986 you worked from home and were working part time as a software diagnostician; is that right?
Anne Chambers: Yes, that’s right. I had first one and then two children and there was a group of people within ICL who were all home based, our management were mostly home based, as well, and, with what now seems like very prehistoric comms and equipment, we could actually do our job remarkably well.
Mr Beer: What does a software diagnostician do?
Anne Chambers: Some – a user of a piece of software that you are supporting somewhere in the world discovers there is a problem with it, at the time they would sort of fill in a paper form saying what the problem was that they’d encountered, and then it was up to me to look at evidence provided. Sometimes it was great heaps of dumps that you had to sort of work out how to read your way through, to try to work out what had gone wrong, identify the problem and, at that time, work out how to fix it, usually by applying something that was called a patch, rather than actually changing the code.
Mr Beer: Just stop there. There’s a noise that I don’t know whether other people can hear. It seems to be coming through the air-conditioning vents. Is it me or –
Sir Wyn Williams: No, it’s not you, Mr Beer. It’s certainly very noticeable where I’m sitting and, for a moment, I thought we were outside in a storm.
If this is troublesome for you giving evidence, Mrs Chambers, we’ll try and do something about it. Is it bothering you?
Anne Chambers: I think I can ignore it, as long as it doesn’t get too much louder.
Sir Wyn Williams: Well, the usher is going to try to make some investigations but we’ll carry on for the moment and see where we go.
Mr Beer: Yes.
In the answer that you just gave, you said that someone somewhere around the world had found a problem with the system. Would a software diagnostician always be responsive to somebody else finding a problem or would they, in some cases, proactively look for code faults, errors or bugs in the software?
Anne Chambers: In these particular instances that I was supporting at that time, it was dependent on somebody reporting the problem to us.
Mr Beer: So it was always reactive?
Anne Chambers: Yeah, it was very reactive.
Mr Beer: So you would investigate error reports –
Anne Chambers: Yes.
Mr Beer: – is that a good way of describing them – filed by users?
Anne Chambers: Yes.
Mr Beer: Would a fair way of describing what you did was produce code fixes?
Anne Chambers: Yes, produce patches to the code, yeah.
Mr Beer: That changed when you went on later to work for the SSC. You did the former but didn’t produce code fixes; is that right?
Anne Chambers: That’s right, yes. I had been doing – it was usually called fourth line support, so people would have already checked for published known errors, and things like that, although sometimes things got through.
Once I moved on to the Post Office work, I was third line support, where we were doing a great deal of the investigation but we would not actually be fixing the problem ourselves, and not necessarily finding the root cause of the problem ourselves.
Mr Beer: Can I look at this early stage, before October 2000, and your early involvement in Horizon. You tell us in your witness statement, paragraph 3 – no need to turn it up – that from 1997 you did some coding and support in respect of part of the new Pathway system for the Post Office; is that right?
Anne Chambers: Yes, that’s right.
Mr Beer: The purpose of the questions I’m about to ask you is to establish what you did learn if you did learn things as a result of your early involvement in the development of what became Horizon; do you understand?
Anne Chambers: Yes.
Mr Beer: In which team were you working from 1997 onwards until October 2000?
Anne Chambers: I was still working for my offsite team. I think we were called ICL Systems at that point. I was still doing – supporting other systems as well as working on the – on this particular niche area of Post Office.
Mr Beer: Who was your manager or supervisor at that time?
Anne Chambers: I think my manager was Sheila Powell. But again, as I say, she wasn’t Post Office. She wasn’t part of the ICL Pathway team. This was still part of this separate structure.
Mr Beer: Were you still home based at this time –
Anne Chambers: Yes, I was –
Mr Beer: – or had you gone back into the office?
Anne Chambers: – I was still home based.
Mr Beer: Did you ever go into the office in this period, 1997 until October 2000?
Anne Chambers: Um, as regarding the Pathway work in that period, I remember going to Feltham, I think, once and I remember giving a couple of training sessions in different locations.
Mr Beer: Concerning the Post Office Benefits Agency project?
Anne Chambers: Yes. Can I just explain this area that I was working on. It was the transfer of files out of the Benefits Agency. There was some transformation done, so that they could then be fed into the back end of the Pathway system.
Mr Beer: I was going to ask you about that because your witness statement suggests you were involved in the Benefits Agency side of the coding?
Anne Chambers: Yes. Well, it was a funny sort of lump in the middle but the Benefits Agency side of it used VME which was ICL’s proprietary operating system, or one of them, and that was what I was particularly an expert in if you like.
Mr Beer: Was it restricted to that?
Anne Chambers: And that was – my only involvement was the transformation that was done on the data in these files and the transfer into the back end of the Post Office system. But I didn’t know any more about what happened to those files and I knew nothing about the counter end of the Post Office system at that time.
Mr Beer: How long did this work on the Pathway system, by you, last?
Anne Chambers: I still had some involvement by the time I joined the SSC because, by that time, there was only about one file left that was being processed. I think it was to do with Child Benefit – I think that’s right – and I was still providing some level of support, for that.
Mr Beer: So from ‘97 until October 2000, all be it doing other jobs for ICL –
Anne Chambers: Yeah.
Mr Beer: – involved in the Pathway system in the way that you explained?
Anne Chambers: I had some involvement yes. Certainly by ‘99 there wasn’t – it was just sort of support of this one file that was being transferred. So I was doing quite a few other things then, as well.
Mr Beer: The Inquiry has heard evidence that there were systemic design problems with the development of Horizon from the outset, including in respect of the integration of Pathway and benefits and Post Office systems, and has heard evidence of problems with the requirement specifications for the project. In general terms, in that three-year period, were you aware of such problems at the time?
Anne Chambers: Only in as much as the vast majority of it was canned and the relationship obviously was – would appear then not to have been particularly good. But, no, I had no direct knowledge of that.
Mr Beer: What was your understanding of the reasons that the majority of it, the project, was canned?
Anne Chambers: Just that the different bodies involved couldn’t work out properly how they wanted it all to work together. I don’t know. I wasn’t involved in any of the political side of it, if you like. I was – have always been very much technical and not involved in the more political and perhaps commercial aspects.
Mr Beer: But you picked up that it was a problem with the parties working together that was the problem; is that right?
Anne Chambers: I think that was the impression I got at the time.
Mr Beer: Did you pick up anything else, that it was a problem with the system or the quality of the Horizon System?
Anne Chambers: No, because I don’t think – I mean, I don’t think I knew particularly what happened to the data that was in the files that we were passing on. So …
Mr Beer: That’s one way that you may have learned, ie feedback on the issues upon which you were working but I’m talking about talking to your colleagues, receiving emails, attending meetings –
Anne Chambers: I don’t recall any of that.
Mr Beer: So was the extent of your knowledge that the majority of the project was canned, that it was to do with a relationship problem rather than technical issues with Horizon itself?
Anne Chambers: I think that was the impression I got at the time, yes.
Mr Beer: Can we look, please, at POL00091901. That should come up on the screen for you. You should see that this is an “Operational Review of the CAPS”, “CAPS” meaning Customer Accounting and Payment System, yes?
Anne Chambers: I can’t remember if that’s what it stood for, but, yes, it could well have done.
Mr Beer: Take it from me. I mean, if you want to look ahead to page 9 of the document and at the foot of the page. “CAPS” in this sense, Customer Accounting and Payment System; can you see that?
Anne Chambers: Yes, I can see that.
Mr Beer: If we can just go back to page 1, please. So “Operational Review of the CAPS/Pathway Interface”.
Anne Chambers: Yeah.
Mr Beer: It’s dated 26 February 1998.
Anne Chambers: Yeah.
Mr Beer: If you look at the distribution list, I think you’re on it.
Anne Chambers: Yes.
Mr Beer: The “O” being Olivia?
Anne Chambers: Yeah.
Mr Beer: We can see that on the right-hand side, second entry.
Can we go, please, to page 61 of the document and look at paragraph 6.3.8.2, the second paragraph down. The document reads:
“Anne Chambers (ICL Systems) has expressed doubt that NEXT-ACTION-TIME can actually be explained convincingly as it is and that CAPS and Pathway should get together to produce a proper definition of the requirement. A definitive specification would provide a basis for reviewing the current implementation as well as a document that would be useful in supporting the Live Service.”
So you’re reported in this document as saying that CAPS and Pathway – is that essentially the Benefits Agency part of the programme and ICL – should get together?
Anne Chambers: Yes, I mean, we’ve got a file or files that are being transformed and passed over and it’s obviously important that both sides agree exactly the definition of the data that one is sending and one is receiving.
Mr Beer: So would this be an example of there being some doubt or ambiguity, whoever’s fault it was?
Anne Chambers: Yes, it didn’t seem as if this had been properly defined because – I can’t remember, I think by the time I got involved the code had already been written, but I think this was a particular field that we were having some problem making sense of exactly what it was meant to contain, and the assumption had been made that it should be like that but it wasn’t clear that that was correct.
Mr Beer: Presumably you’ve got no memory of this now?
Anne Chambers: No –
Mr Beer: No.
Anne Chambers: – very, very vague memories and I certainly couldn’t tell you how it was and how I thought it should be or anything like that.
Mr Beer: No, but the issue of there not being a proper definition of the requirement, that’s the customer’s requirement, yes?
Anne Chambers: Yeah.
Mr Beer: Can you recall whether that was, in this sort of three-year period, something that happened often or a recurring issue?
Anne Chambers: I don’t remember that, no. I think – I mean, it would only be at the point that these files actually started being – using the code that had been produced and once they’re actually being processed, um, that then you’d have some debate about whether these things were actually as both parties had understood.
Mr Beer: Can we go forward to page 75, please, and look at, under the heading “Question 3: Adequate Resilience”, 7.1, “Statement of the Question”, the question is:
“Is the operation of the interface adequately resilient in terms of its ability to recover from failure states?”
Then if we go down to 7.3.1.1, “Description of the Issue”:
“In order to pass a file to CAS(VME), the CAPS software writes the file to a CAPS Out Tray and passes a File Notification … to CAS(VME) via XPERT. Certain problems in the use of XPERT have resulted in …”
Then there’s a description:
“When resolving such problems, it has proven very useful to be able to pass a File Notification to CAS(VME) manually. This has been done by using CAS_MEND, which was provided informally by Anne Chambers (ICL Systems), a member of the CAS(VME) development team. It is anticipated that similar problems will be encountered in the future and that the same SCL procedure or something very like it would prove equally useful.”
There is no need for us to explore the technical details of what is being spoken about there but is what is being described the fact that you had yourself developed and provided a workaround utility?
Anne Chambers: Yes, I think it was possibly something that we’d had done for our own testing. Obviously, when you’re testing things you have to pretend that things are happening, to some extent, and it turned out that, you know, there was some sort of a requirement for this. That first paragraph is all stuff that was very much in the Benefits Agency camp.
Mr Beer: Yes. If we can go over the page, please – to 77, sorry – the report continues:
“However, it was pointed out that the condition that gave rise to actions, in which this utility was used, was an error condition and not normal processing. Such an error condition should be investigated and understood, the current situation recovered, and the root cause eliminated to prevent repetition. Therefore, occasion for the use of the utility should be very rare indeed.
“It was further pointed out that the use of the utility affects audit data for CAS(VME). The CAS ICL is updated with information from a File Notification specially created on the CAS side of the interface. That information is passed forward to the ICMF. It was queried whether, in principle, a utility of this nature should be provided by Pathway as a standard component of the CAS(VME) product, since it compromises the integrity of the audit trail and its use could provide an embarrassment to Pathway in any contractual dispute.
“A compromise position was formulated. It was recognised, by Pathway and CAPS, that the interface is not yet fully stable and that problems of the kind described may be encountered in the future. Such problems require that there should be a means of recovery.”
Is what we’re seeing described here evidence that fixes designed to address errors could themselves impact – I’ll just stop and start the question again.
Is what we see here evidence that fixes designed to address errors could themselves impact on audit trails for the systems being developed?
Anne Chambers: In theory, yes, they could and you wouldn’t be using something like this unless you absolutely had to. It shouldn’t be a standard way of doing things if it then couldn’t be audited or whatever.
Mr Beer: Why was that? Because this was sort of an ad hoc fix developed by you?
Anne Chambers: It was – it wasn’t developed as a fix. It was something that existed that we could use, and I think it was initially for – possibly for testing. I don’t think it was called CAS_MEND originally but it was something so we could put an entry in a table to say “Look, here’s a file”, to get over this error condition but that shouldn’t have ever been a long-term fix to the problem.
But sometimes if you had to choose between doing something like that that would then have to be documented as an unscheduled sort of a change, if you could either do that or a whole day’s benefit – Child Benefit payments couldn’t go through, then that’s something that has to be weighed up against each other.
Mr Beer: Why would it compromise the integrity of the audit trail?
Anne Chambers: Well, that’s what it’s suggesting here, isn’t it?
Mr Beer: Yes, but why would it compromise the integrity of the audit trail?
Anne Chambers: I cannot now remember enough about the details to say.
Mr Beer: Would the integrity of the audit trail be an important principle to maintain?
Anne Chambers: Yes, it always is.
Mr Beer: Why is that?
Anne Chambers: Because then if there are questions afterwards about something, you need to be certain that you have got a proper record of what was done.
Mr Beer: You will see that it mentions the report – the fix compromising:
“… the integrity of the audit trail and its use could prove an embarrassment to Pathway in any contractual dispute.”
Can you assist as to why the use of what I’ve described as the fix could prove an embarrassment to Pathway in a contractual dispute.
Anne Chambers: I cannot remember enough about all of this to be certain but there was obviously record kept of the files that had been received and the sizes and the dates and all that sort stuff, which would have been, I presume, part of the audit trail, and I can’t be certain now but from what this is saying, it suggests that however we were notifying the system that there wasn’t another new file had come in, but the notification wasn’t arriving in the normal way. That can’t have been recorded in the normal way, I presume.
Mr Beer: Can you help us more broadly – that document can come down, thank you. Did you hear any word amongst your colleagues or chatter or similar, about how the Pathway Project had gone for Fujitsu by the time you joined the SSC in October 2000?
Anne Chambers: Um …
Mr Beer: What was the word on the street?
Anne Chambers: I knew that at that point the rollout was going ahead. I think when I started there were about 25 per cent of Post Office branches had got the new Horizon System and so, obviously, it was ramping up very rapidly and I certainly – I can’t remember. I don’t recall anybody saying it was so dreadful enough to make me feel I did not want to be a part of it.
Mr Beer: What about something less than that? Were you told, for example, when you were joining the SSC or beforehand, that a range of problems and issues had been encountered in the design, build and rollout of Horizon?
Anne Chambers: No. I mean, you would expect there to be a certain level of problems and they obviously needed more people in SSC. There was quite a lot of recruitment going on which, by the nature – you know, that is group of people who are providing support. So there was obviously a need to have that group and to build it up. But I didn’t feel – I wasn’t aware of anything, you know, “Oh, this is so bad we’ve got to have so many extra people on it”. It was, you know, “This is an exciting new project, it’s at last, after many years of preparation, it’s up and running, great, let’s keep it going and make sure it’s all working well and doing its job”.
Mr Beer: Were you told that the Benefits Agency had pulled out because of concerns over the integrity of the data that Horizon produced?
Anne Chambers: No.
Mr Beer: When you joined the SSC, did you therefore think you were to be providing support for a good and properly functioning system?
Anne Chambers: I anticipated that it would have problems, otherwise there would have been no job for me to do there.
Mr Beer: Yes, that doesn’t really answer the question, Mrs Chambers.
Anne Chambers: No, I don’t think anybody in – who’s doing computer support work ever sort of – you know, the whole purpose of our existence was to get on top of any problems that there were, and this is probably going to come out wrong but, in some ways, the whole – not exactly enjoyment of the job but what you’re there for is to sort out these problems so you do anticipate that, yes, there will be things to get your teeth into, if you like.
Mr Beer: But were you approaching this that this was just another project in a line and that there was nothing – you weren’t walking into a project that had had a particularly problematic birth?
Anne Chambers: No, that was not how I saw it. I was – for me, personally, I was ready for a change and it was quite a big change because, at that point, I went back on site, I hadn’t actually had to work with other people very much for 15 years, and I was moving from being very technical, doing a fourth line support job, to being less technical. I was also moving away from supporting things on VME, which was my main technical speciality, to something that was using – well, it wasn’t VME-based at all, apart from this one file that was left.
Mr Beer: So you joined the SSC. What did you understand SSC to stand for?
Anne Chambers: Err –
Mr Beer: We’ve had three variants of it.
Anne Chambers: Yeah, System Support Centre.
Mr Beer: Thank you. You joined in October 2000?
Anne Chambers: Yes.
Mr Beer: You stayed there for the rest of your career with Fujitsu?
Anne Chambers: Yes.
Mr Beer: What was your job title when you first joined the SSC in October 2000?
Anne Chambers: I think it was system specialist but I cannot be entirely sure. Job titles did change here and there. They didn’t necessarily – they were usually sort of fairly vague but I think I was a systems specialist.
Mr Beer: Were you now working full time when you –
Anne Chambers: Yes –
Mr Beer: – moved to the SSC?
Anne Chambers: – I had been – I think I’d been working 30 hours plus quite a lot extra from home and so now I was officially 37 hours a week.
Mr Beer: Did you now work in the office?
Anne Chambers: Yes, I did.
Mr Beer: Was that in Bracknell?
Anne Chambers: Yes, it was.
Mr Beer: When you joined the SSC who was your manager or supervisor?
Anne Chambers: Mik Peach.
Mr Beer: But he didn’t remain your manager for the entirety of the 16 years that you worked in the SSC; that’s right, isn’t it?
Anne Chambers: That’s right, yes.
Mr Beer: But when he worked there, to whom did he report?
Anne Chambers: I can’t remember. It was different people at different times.
Mr Beer: Did you report to a director?
Anne Chambers: I don’t – oh. I don’t think so, no. I think there was several layers but I – again, I – my interests were technical and not particularly in the structure of the organisation.
Mr Beer: Did you ever report to the person above Mik Peach or did you always report into Mik Peach?
Anne Chambers: I always reported into Mik or his successors.
Mr Beer: After Mik Peach left, you say in your statement in about 2010 – just for the transcript, Mr Peach says it was in September 2009 – you say that he was replaced by Tony Little for a few months?
Anne Chambers: That’s the name I think I remember.
Mr Beer: And then by Steve Parker?
Anne Chambers: Yes.
Mr Beer: Did Steve Parker remain your manager until you left in 2016?
Anne Chambers: Yes, he did, although we had team leaders as well, so we did have an extra layer.
Mr Beer: Those team leaders, were they introduced by Mr Parker?
Anne Chambers: Yes, from the existing team.
Mr Beer: Were there four teams?
Anne Chambers: I think there were four. I can’t be quite certain.
Mr Beer: Can you remember what the division within the teams was – between the teams?
Anne Chambers: They were just sort of purely for administration, it wasn’t for – it wasn’t sort of one team supporting one particular area or anything like that.
Mr Beer: So there wasn’t specialism –
Anne Chambers: No.
Mr Beer: – team A, specialism team B?
Anne Chambers: No. Except possibly – at some point, the Reference Data Team sort of merged into SSC, and I can’t remember now if they stayed as more or less a separate team or if they ended up reporting to different team leaders.
Mr Beer: So there was sort of a mixed economy of skills within your team –
Anne Chambers: Yes.
Mr Beer: – even though, as we’re going to discover in a moment, you specialised?
Anne Chambers: Yes.
Mr Beer: So to going back to the beginning then in October 2000, there was essentially a flat structure with one manager, Mik Peach?
Anne Chambers: Yes.
Mr Beer: How many people worked in the SSC at that time when you joined?
Anne Chambers: I think it was around 25 but I can’t be certain of that.
Mr Beer: Were they all what I’m going to call diagnosticians?
Anne Chambers: Yes, I think that’s true to say.
Mr Beer: There was an administrator as well on top; is that right?
Anne Chambers: Yes, there was an administrator and then, at one point, an administrator’s assistant as well, and then no administrator.
Mr Beer: What was the function of the administrator?
Anne Chambers: Um, order the stationery; answer the door, because it was a secure unit so people had to be let in; answer the phone; and monitor the stack of service tickets, peak calls coming in and allocating them to members of the team.
Mr Beer: So they had a role in allocation of the PinICLs and then the PEAKs?
Anne Chambers: They allocated them, yes, and also she’d look at any KELs that had been mentioned to see if it looked at a fairly superficial level, if it looked as if it was the right one. If there was absolutely no information on the call giving any clue as to what the problem really was, then she might return the call to second line and ask them to get some more information.
Mr Beer: Was that person the same throughout this period? Was it Barbara Longley?
Anne Chambers: It was Barbara Longley until she retired and I cannot quite recall when that was. I think it must have been before 2008, I think, or 2009.
Mr Beer: How did she determine to whom to allocate a PinICL or PEAK?
Anne Chambers: Um, partly what sort of area it was, um, somebody who hadn’t got any calls on their stack already, obviously –
Mr Beer: So workload?
Anne Chambers: – it would be a – workload, interest. Sometimes somebody – because we could all see this stack of calls – sometimes somebody would say, “Oh, I’d like that one”, or, you know, somebody might point out to her that it was relevant to something else that had already come in.
Mr Beer: Were there specialisms within the 25 of you?
Anne Chambers: Yes. We all – everybody seemed to gravitate to different areas.
Mr Beer: Was that it, the force of gravity, ie personal interest, or was it anything more formal than that?
Anne Chambers: Um, it was partly what people’s backgrounds were when they came in. Um, if Mik felt there was a bit of a gap somewhere and not enough people specialising in one particular area he’d obviously get somebody in and say “Right, you know, you’re doing this”.
Mr Beer: It’s right, though, that you were each expected to handle any type of –
Anne Chambers: Yeah.
Mr Beer: – ticket, if necessary?
Anne Chambers: Yes, we were.
Mr Beer: I think the number of 25 decreased over time; is that right? You tell us in your statement that, by the time you left in 2016, the number had decreased to between 12 and 15 people?
Anne Chambers: I think so. It’s really hard to remember definite numbers, especially because, towards the end of that time, partly we were taking on some extra bits of workload, non-Pathway stuff, some other teams that had been elsewhere in Pathway were now either part of SSC or at least sharing the same floor space as us. So it’s a little difficult to remember who was where and which team.
I’d also say that I think the numbers reduced a bit before HNG-X and then I think we got more people on board then when the new system was rolled out everywhere in 2010.
Mr Beer: So decreased before Horizon Online?
Anne Chambers: I think it had dropped a little bit naturally, just by people leaving and not so many new people coming in.
Mr Beer: You tell us in your statement that you were most likely to deal with tickets that concerned counter balancing?
Anne Chambers: Yes.
Mr Beer: How did that come about?
Anne Chambers: Er, I think largely because I was sitting next to somebody who was an expert in that area and, although she hadn’t been my sort of official mentor when I started, I picked up on a lot of the stuff that she was doing and also, I liked the – you know, playing around with numbers and checking that things added up.
Mr Beer: You say that there were five or six of you, when there were 25, that would be most likely to handle tickets that concerned counter balancing; is that right?
Anne Chambers: Probably, yes. I mean, more of us would have – there was certainly a lot of other people who might occasionally have picked up a call of that type but probably the more complicated problems would come down to, you know, five or – four, five or six of us.
Mr Beer: Can you remember who they were?
Anne Chambers: Um, yes. I mean, Diane Rowe early on; Dave Seddon and Lina Kiang, who were both there for longer than I was; Sudip Sur who started at about the same time as me; Cheryl Card, who started later; and then people like John Simpkins and Mark Wright, who knew a great deal about everything, wouldn’t maybe be doing those sort of calls so often but they had a very good knowledge of the entire system, and I apologise to anybody I’ve left out of this.
Mr Beer: Did your role in counter balancing mean that you became a specialist in the operation of Riposte and the EPOSS system?
Anne Chambers: Well, we all needed to know a lot about Riposte anyway because it was at the heart of the entire system but yes, the EPOSS system, I would really perhaps – where I’ve talked about counter balancing, I mean, a lot of the problems were more general EPOSS, counter front end part of the system.
Mr Beer: You’ve told us that this specialism developed because of the person that you were sitting next to.
Anne Chambers: Mm-hm.
Mr Beer: Can I just explore with you what, if any, training you had on and about the Horizon System before you became responsible for investigating problems and issues with it and the integrity of the data that it produced. You tell us in your witness statement that, in 2000, you and some other new joiners attended the same counter training that was providing for subpostmasters; is that right?
Anne Chambers: Yes, that’s right.
Mr Beer: How long did that counter training last?
Anne Chambers: Um, I think it was probably a week session and it was a course run especially for us just in a room on our secure floor.
Mr Beer: Was the training, to your knowledge, in any way changed because you were the system diagnosticians or were you treated as if you were subpostmasters?
Anne Chambers: I think we were treated as subpostmasters because it’s useful to see it, you know, from the end user’s point of view. Although, obviously, we didn’t have the business knowledge that any postmaster who’d been running his branch using the paper systems for years, they would come in with that sort of knowledge.
Mr Beer: In the course of that training, were you told about concerns, issues or defects in the Horizon System?
Anne Chambers: I don’t recall being told of any during that training.
Mr Beer: Now, the counter software used for balancing was maintained by the EPOSS system within development, the fourth line support; is that right?
Anne Chambers: Yes.
Mr Beer: Did you know at this time, on joining or shortly there afterwards, any internal reputation within Fujitsu of EPOSS during the development of Horizon, that it had been rather problematic or troublesome?
Anne Chambers: I don’t recall that, no.
Mr Beer: So, again, you were thinking you were operating a system that was well oiled and functioning but may turn up problems because, otherwise, you wouldn’t have a job?
Anne Chambers: Yes. I think that’s true.
Mr Beer: Can we look, please, at WITN04600104. This is an ICL Pathway report dated 10 May 2000, you can see that on the top right, so a few months before you took up your post, yes?
Anne Chambers: Yes.
Mr Beer: It concerns the results of an audit. You’ll see that it’s titled, both at the top and in its first line, “Schedule of Corrective Actions, CSR+ Development Audit”. Now, if we scroll down we can see that you’re not on the distribution list and I’m not suggesting that this was shown to you in any way.
Can we go to page 9 of the document, please, and can we look, please, at the first column in the table:
“The audit identified that EPOSS continues to be unstable. PinICL evidence illustrated the numbers of PinICLs raised since the 1998 Task Force and the rate of their being raised.
“The EPOSS Solutions Report made specific recommendations to consider the redesign and rewrite of EPOSS, in part or in whole, to address the then known shortcomings. In light of the continued evidence of poor product quality these recommendations should be reconsidered.”
Did you know, when you joined the SSC, that an audit of the EPOSS had found it to be unstable?
Anne Chambers: No.
Mr Beer: Did you know that a report had concluded that EPOSS should be redesigned and rewritten?
Anne Chambers: No.
Mr Beer: Did you know that in May 2000, a few months before you joined, that that recommendation had been repeated?
Anne Chambers: No.
Mr Beer: Can we go to page 10 of the document, please, and look at the response. It’s in the bottom right-hand corner. Thank you:
“Following response received from MJBC: ‘As discussed this should be closed. Effectively as a management team we have accepted the ongoing cost of maintenance rather than the cost of a rewrite. Rewrites of the product will only be considered if we need to reopen the code to introduce significant changes in functionality. We will continue to monitor the code quality (based on product defects) as we progress through the final passes of testing and the introduction of the modified CI4 codeset into live usage in the network. PJ can we make sure this is specifically covered in our reviews of the B&TC test cycles. Closed.”
Did you know, when you joined, that the quality of the EPOSS code, based on, as there described, product defects, was supposed to remain under review during the introduction of the modified codeset into live usage in the network?
Anne Chambers: No.
Mr Beer: You were part of the SSC in the months following this report. To your knowledge, were people, including you in the SSC, told of the need to monitor the EPOSS code through product defects?
Anne Chambers: I don’t recall being told that, and it’s perhaps something that I would have expected our manager to have been keeping an eye on, rather than – I mean, because he knew all the problems that were coming in, rather than of us – certainly people who have only just started, who will just be looking at individual incidents as they happen.
Mr Beer: Is that in fact the case: that he would look at every ticket and see the outcome of it?
Anne Chambers: I cannot speak for him but I think it’s – he certainly had the ability to do that.
Mr Beer: The ability, yes, but, to your knowledge, in the 16 years that you worked there, did the manager perform that kind of function? There’s a recommendation here that this action be closed, that there be no rewrite, no redesign of EPOSS because there’s going to be a monitoring process?
Anne Chambers: Yes, but I wouldn’t expect something like that to be monitored by the people, if you like, at the very pot of the heap. I would have expected somebody slightly higher up, for example the SSC manager. But I obviously cannot say “yes” or “no” he did this. I think, knowing Mik, its quite likely that he did, but it might have been him, it might have been somebody else on –
Mr Beer: If you didn’t know about this, you wouldn’t know to feed back “I’m noticing a preponderance of problems with the EPOSS system or the code in this part of the EPOSS system”, would you?
Anne Chambers: No, I wouldn’t but then, as I said, I would have expected that to have been monitored at slightly higher level.
Mr Beer: Would you expect the people at the lower level, as you called it, including yourself, to have contributed to that, ie a monthly review or a quarterly review or even a yearly review: let’s look at how EPOSS is performing?
Anne Chambers: Um, I don’t know. I mean, no, I still feel that’s the sort of thing that, you know, where you’ve got a lot of people working not exactly individually, but when the information is all there on the PEAKs, and so on, I would have – I think it seems much more likely and sensible, in some ways, for it to be looked at by somebody who’s got the technical knowledge but has – you know, their job is to take the broader view –
Mr Beer: But there wasn’t any formal instruction to you or informal instruction to you to say, “Chalk up when you’re dealing with a ticket, a problem with EPOSS” –
Anne Chambers: No.
Mr Beer: – “so that it can be fed back to somebody conducting an overarching review to carry this recommendation into effect”?
Anne Chambers: No, we were never told to do that.
Mr Beer: When you joined the SSC, what was the role of Gareth Jenkins?
Anne Chambers: I was aware that he was one of the technical experts. I think to start with, he was – I’m not sure if he was based in Feltham then, where a lot of the development teams were, but I don’t think I met him for – until I’d been there for two or three or four years.
Mr Beer: So 2003, 2004?
Anne Chambers: Possibly. It might have been slightly sooner. I think I became aware of the name because you saw it on documents, and so on. But SSC were very much self-contained on our floor because it was a skill floor so you didn’t have people coming and going, so we sort of, to quite a large extent, kept ourselves to ourselves.
Mr Beer: Did you understand him to be the principal Fujitsu expert on the counter application?
Anne Chambers: I probably picked that up fairly quickly, yes. I don’t think anybody ever told me that.
Mr Beer: Was there any process of induction to say, for example, “This is Mr Jenkins, he’s the chief designer/architect of, I don’t know, the changes to POL’s back end systems, that meant he works a lot with the counter application and the EPOSS code”?
Anne Chambers: No, I –
Mr Beer: “If you have [X] problem, he’s your point of contact”?
Anne Chambers: No, and he wouldn’t, at that point, necessarily have been our next point of contact because we would probably have talked to the EPOSS developers about any problems in the first instance and then I’d have expected them to go and talk to Gareth if necessary.
Mr Beer: By the EPOSS developers, do you mean people in-house?
Anne Chambers: Yeah.
Mr Beer: Did you form any opinion of the quality of the EPOSS developers?
Anne Chambers: Um, I’m trying to think who was there. Yes, I didn’t work closely with them. As I said, to start with, they were in Feltham anyway. I think there’s always a slight tension between support and developers, who are also doing support, because they are often actually developing enhancements to the system, and so on. And so sometimes, perhaps you’ve felt you wanted them to focus a little bit more on the support of an existing problem but they were heads down working on something new.
Mr Beer: In the months after you joined, did you form a view on the quality of the product, the EPOSS, that they were working with?
Anne Chambers: Um, I don’t think I thought of it in those terms at that point. You know, this was what we were looking after. We dealt with whatever came up and, where necessary, we passed things on to EPOSS. I can’t remember in those very early days – when things were still potentially settling down after the rollout, the only thing that I can remember is that there were – I can remember one call in particular to do with a cash account production, where it was very difficult to get to the bottom of the problem and to work out what the numbers on the cash account should actually have been, and so on, and I think there was someone called Steve Warwick, who I think was involved in trying to help out with that.
So I remember that as just one particular call where there was a particular problem and difficulty and I cannot remember what the root cause of it was.
Mr Beer: But you didn’t have an overarching view of EPOSS, that it was a problematic or troublesome system? The Inquiry has heard some evidence already, in its Phase 2, as to the views of some of those within Fujitsu and Post Office –
Anne Chambers: Mm-hm.
Mr Beer: – as to the quality of the EPOSS system, one describing it as “a bag of” and then an expletive. When you took over in the SSC, it didn’t strike you as being deeply problematic?
Anne Chambers: No, I mean, by this time, there were, I don’t know, perhaps initially 10,000 Post Office Counters using it every day for all their business, and then 15,000, and then 25,000, and finally about 37,000 counters using it, and, although yes, obviously, some calls were coming in and some of them were EPOSS, we certainly weren’t being swamped with the number of calls that you would expect if the system was thoroughly rotten, because it just – you know, once you’ve ramped up to those volumes, you are going to – if there are problems, you are going to be seeing them.
Mr Beer: Assuming they made it to the third line support?
Anne Chambers: Yes, but, basically, you know, this did seem to be a usable system because it was being used.
Mr Beer: You mean because it didn’t fall over?
Anne Chambers: It didn’t fall over. People weren’t reporting, “Oh, I’ve pressed this button to sell a First Class stamp and it’s sold”, I don’t know, something else instead. We weren’t getting large numbers of calls from people saying, “Oh, we did this and it’s not there”, and so on.
So I think it’s – you know, it’s hard to put it into words, but we weren’t getting, if you like, the feedback from the live estate that it – that there were a huge number of significant problems.
Mr Beer: So these fears that had been expressed, just months before you joined, that there needed to be a total redesign and total rewrite of EPOSS, when the system was working, they just didn’t come to pass?
Anne Chambers: Well, it may well be – I don’t know, you gave the date on the front of this as being –
Mr Beer: 10 May.
Anne Chambers: Yes, but that was the final edition of that document rather than when it was initially written?
Mr Beer: Correct.
Anne Chambers: So it’s quite possible that bug fixes and other changes would have been made to the system in that period. So, you know, the system wasn’t static, things were being fixed and enhanced, all the way through its life.
Mr Beer: The Inquiry understands that a gentleman called Matt Aris, A-R-I-S, was the EPOSS development team leader; do you remember that?
Anne Chambers: I remember the name.
Mr Beer: Do you remember him being the development team leader?
Anne Chambers: I couldn’t have sworn to that if you hadn’t just told me.
Mr Beer: Can you help us: what would be his, if he was the development team leader, his relationship to Gareth Jenkins?
Anne Chambers: I assume that if there was – when changes to the system were – when changes to the code were happening or to the design, he would use Gareth to discuss anything that needed discussing, and so on.
Mr Beer: So he was more senior to Mr Jenkins?
Anne Chambers: No, Mr Jenkins would have been more senior, I would have thought.
Mr Beer: Were they in the same team, the same reporting structure?
Anne Chambers: I’ve no idea.
Mr Beer: Did you have dealings with Mr Aris?
Anne Chambers: I almost certainly talked to him. I think I did talk to him. To start with, as I said, we were quite a self-contained team and, if we wanted to pass a ticket on to fourth line because we thought there was a code problem and they needed to investigate further, then the way of doing that was just to assign it on PEAK, so it got passed through.
As time went by, I have always liked to try to develop some sort of relationship between teams and so, certainly, once the development teams had moved into Bracknell, then I would quite likely walk down a flight of stairs and go and talk to them about something, rather than just saying, “Oh, well, it’s off my desk”, and passing it on to them in that way.
Mr Beer: Can I turn, before we have the morning break, to the ways in which the SSC operated in practice. I’ve got ten or so issues I want to ask you about, please:
Firstly, the data available to you.
Secondly, the process by which tickets were passed to SSC and, in particular, the system for linking them in to a KEL.
Thirdly, concerns about the SSC fobbing off subpostmasters.
Fourthly, how the SSC would go about establishing the extent of a problem when it received a ticket.
Fifthly, what information was passed back to subpostmasters by the SSC or others.
Six, some other problems with the PEAK system.
Seven, the process of pacing an investigation around a single PEAK.
Eight, looking at the Horizon Helpdesk role.
Ninth, the use of ARQ data.
Tenth, attributing a problem to user error.
Okay, so they’re the ten topics we’re going to look at.
Firstly, then, the data available to you when a ticket was allocated to you.
You tell us in paragraph 30 of your witness statement, maybe if we can turn that up, please. Witness statement, paragraph 30, which is on page 8. You tell us in the last sentence of the main part of paragraph 30:
“In relation to counter issues for Legacy Horizon, the primary sources of evidence would be …”
Then you set out three bullet points. So the first one, is that the branch data in the message store?
Anne Chambers: Yes, this is all the branch transaction data and various other messages that would be written to the message store as well and all the reference data for the branch.
Mr Beer: Just now, for later on when I ask you questions, it’s right, is it, that that that, could later be retrieved from an archive via Fujitsu Security and is referred to as the ARQ data?
Anne Chambers: Yes.
Mr Beer: Yes? Is that a shorthand summary?
Anne Chambers: Um, yes, I mean, the ARQ data could either contain the whole of the message store or – well, it was a slightly – I don’t know how I can explain this without explaining a bit more about message stores and Riposte but you may not want to go into that now.
Mr Beer: I probably don’t, thank you.
Anne Chambers: Okay.
Mr Beer: But, for present purposes, it’s sufficient to note that this first bullet point contained data that was archived?
Anne Chambers: That data was all archived, yes.
Mr Beer: Fujitsu Security could access it and a way of describing it is ARQ data?
Anne Chambers: Yes.
Mr Beer: Okay. Then, secondly, the event log from the Horizon counter application?
Anne Chambers: Yeah.
Mr Beer: Then, thirdly, the –
Anne Chambers: Sorry, could I go back to the second one. That’s actually the Windows NT application event log, so it’s not just the Horizon application that’s writing to it.
Mr Beer: Okay, can you just describe, for the benefit of those listening, what the Windows NT log was, then?
Anne Chambers: Any events that have been generated by an application running on a computer or by the Windows system itself would be written to this event log.
Mr Beer: So, essentially, events in the Windows product that the counter application was built on top of?
Anne Chambers: Yes, but also counter application events as well would be in there. But it’s not purely counter application events. There would be events from other processes running on the counter, as well.
Mr Beer: Then, thirdly, the psstandard.log from the counter. Can you explain what that is, please?
Anne Chambers: That that was – I think “ps” stood for “peripheral server” but it got written to by various things, so in that we could see stuff like what had been output on the tally roll printer at the branch, and so on. There was also a certain level of diagnostics came out somewhere, and I can’t remember if they were also in the psstandard.log or if I’ve missed something and they went somewhere else.
Mr Beer: So the two event logs you mention in the second and third bullet points there, on which servers were they stored?
Anne Chambers: They weren’t stored on servers; they were only stored on the counter.
Mr Beer: They weren’t stored on servers at all?
Anne Chambers: The logs – the events were sent to the data centre through something called Tivoli, I think, and then they were stored.
Mr Beer: Where were those servers?
Anne Chambers: At the data centre, one in Bootle and one in Wigan, but I couldn’t tell you the names of the particular servers that these were stored on.
Mr Beer: Were there back-up arrangements for those servers?
Anne Chambers: Almost certainly but I don’t know any of the detail.
Mr Beer: You can’t help us with what those back-up arrangements might have been?
Anne Chambers: No, and I don’t think that the stream of events, although it was there for monitoring, and in fact they were saved for posterity, they weren’t sort of securely locked and audited in the way that the message store data that could then be retrieved via an ARQ request was locked and kept.
Mr Beer: That was my next question. What processes were employed to ensure that the data on those two event logs was archived and maintained securely?
Anne Chambers: I don’t think it particularly was.
Mr Beer: You said in an answer before last that they were just kept for posterity.
Anne Chambers: Mm.
Mr Beer: By that, did you mean by accident, as it were, rather than by design because the archived data might be needed?
Anne Chambers: Yes, I think it was more that a lot of files were kept for quite a period. But data that was intended for future use in prosecutions, and so on, if you like, was – that was very carefully secured and then there were sort of proper ways of accessing it, and so on.
Mr Beer: But that process wasn’t extended to the data archived in relation to these two event logs, have I understood you correctly?
Anne Chambers: The application event log, no, and the psstandard logs, they didn’t go anywhere except they were just on the counter, so we could retrieve them, and they were only there for quite a short period of time.
Mr Beer: So when you and the SSC retrieved data from event logs and including from the archive, how was that process recorded?
Anne Chambers: I don’t think it was. We wouldn’t – the long-term event archive was very rarely used. We didn’t – I didn’t know it was there until 2006. The stream that went through Tivoli we could look at and I cannot remember if that had anything behind it that did secure that for any length of time.
If we pulled an application event log direct from the counter or the psstandard.log direct from the counter, I’m not sure that was recorded anywhere that we had done that.
Mr Beer: Was nothing done to ensure that the retrieval of data from these two sources was recorded and was undertaken in a secure, auditable way?
Anne Chambers: I don’t think it was, no. It was the only – the security about it was that we were in a locked floor with fairly restricted access to the counters.
Mr Beer: On the counter application, what sort of events would be recorded?
Anne Chambers: Um, the one that springs to my mind is if Riposte outputs – Riposte being not part of the counter application but underlying it – if that produced an error, or even just – you’d also have startup messages in there, so as the counter application started up, it would write various events saying where it had got to in the process.
Mr Beer: Who programmed the counter application to record which events?
Anne Chambers: I presume it was in the development code but I’ve no idea.
Mr Beer: Do you know the decision making that had been applied into which events were recorded and which were not?
Anne Chambers: No. These are not – there’s a big opportunity for misunderstanding here. The counter application itself wrote events into the message store to say when somebody logged on and logged off or when they did a declaration or when they produced a report. Those sort of events. But those are very Riposte events stored in – sorry, not Riposte events. Well, they’re events that are stored in the message store rather than in the application event log.
Mr Beer: On the NT event log –
Anne Chambers: The NT event log.
Mr Beer: – that was presumably a result of Microsoft programming?
Anne Chambers: No, the counter application, if …
(Pause)
Anne Chambers: Yes, I don’t think I remember well enough to explain this. If I had an example in front of me, I could probably work through it and explain things to you but, trying to remember it cold, I don’t think I’m going to be able to add a lot more here.
Mr Beer: If you investigated the event logs whilst dealing with a ticket, would you preserve the event logs with the ticket, ie with the PEAK, or alternatively in the KEL, or not preserve them at all?
Anne Chambers: If the ticket needed further investigation and was going on to fourth line, then, yes, the event log would be attached to the PEAK, along with the message store, and anything else we’d found that looked useful because the SSC were the only team who could get this information out of the live system, so we were expected to get what we could because then that was all that fourth line support would be able to look at to try to find the root cause, and so on.
If our investigation didn’t find anything further that was needed, for example it was another instance of a known error or something else, then these probably wouldn’t be saved.
Mr Beer: If they weren’t preserved in the way you’ve just described, how long was each species of event log retained for?
Anne Chambers: That would be up to the individual. I would probably keep everything I’d looked at for at least a year, if not longer, just in case there was any follow-up.
Mr Beer: But that was a matter of individual discretion amongst the 25 of you?
Anne Chambers: Yes.
Mr Beer: Where would you keep it?
Anne Chambers: On our secure server.
Mr Beer: So what would you do? Would you save it as a file?
Anne Chambers: Yes, it would be saved and when we extracted it, it would go into somewhere in our own area –
Mr Beer: So almost saving to desktop?
Anne Chambers: Not on our desktop, no, on a remote server that we had access to.
Mr Beer: Why did you settle on to a year to keep?
Anne Chambers: Sometimes it would be longer. If I felt I was starting to run out of space, I would – I would very occasionally do a tidy-up but I wasn’t the tidiest person in the world.
Mr Beer: But it was down to your individual discretion?
Anne Chambers: I believe so, yes, I don’t think anybody ever said, “Oh, you must keep this”. I’m sure nobody ever said.
Mr Beer: In addition to the data that we just looked at, when a ticket was assigned to you, if appropriate, you would have had a KEL, yes?
Anne Chambers: If somebody who had already looked at it at first or second line, or potentially the pre-scanner, had decided that a KEL – an existing KEL looked applicable –
Mr Beer: Looked vaguely relevant?
Anne Chambers: – then, yes, they would have put a mention to that on the PEAK or the PowerHelp call and then it was just a hotlink to click on it and to read the detail.
Mr Beer: If they hadn’t made that association, would you nonetheless check the KEL system to see whether there was one?
Anne Chambers: Um, probably, yes. That was probably the process. In practice, once I had been there for some length of time, if it was a call, an incident coming in about something that I was already familiar with, I – you know, I might well know without the searching which KEL it was. But, yes, certainly if something came in, somebody reporting a particular error message, then you’d do a KEL search for that error message or whatever and, if you found something, then that’s your starting point.
Mr Beer: How would you do a KEL search?
Anne Chambers: Um –
Mr Beer: Was it a free text keyword search?
Anne Chambers: It was a very, very free text search, so you just entered a few words that you thought might be relevant. Obviously, if you’ve got an error number or something like that, that’s a good starting position, or an event from a particular source, there would be clues in that as well. So you could type any or all of these things in and see what you’ve got.
Mr Beer: How accurate and reliable was that process in turning up relevant KELs?
Anne Chambers: Pretty good but, like any system, it depends how well they’ve been written in the first place. But certainly for something like a specific error number, yes, if there was a KEL, you were very likely to find it.
Mr Beer: Then, lastly before the break, you also had the databases of past PinICLs and PEAKs, is that right, that you could access?
Anne Chambers: Yes. Again, that was a free text sort of a search, I think.
Mr Beer: I was about to ask, how would you search the database of PinICLs and PEAKs?
Anne Chambers: Yeah, I’m trying to think back. Certainly, by the time left, I’m just about certain it was very easy to search. Again, a free text search.
Mr Beer: Would you habitually do that? If a ticket came in, would you go to the PinICL and PEAK database and look at that database to investigate the current ticket?
Anne Chambers: I’d be more likely to do it from the KEL system.
Mr Beer: So only if there was a link to past PEAKs or PinICLs in the KEL, would you click the hyperlink through; is that right?
Anne Chambers: Yes, probably that would be the normal way of doing it.
Mr Beer: Yes, thank you very much. I wonder whether that’s an appropriate moment.
Sir Wyn Williams: Yes.
Mr Beer: Just in relation to the noises, the first noise was a waste disposal unit’s pistons needing oiling. That has been done. The second noise was a mobile phone and that won’t happen again, I’m sure. The third noise was a fire alarm not in this building because we wouldn’t be here. It was of an adjacent building behind us, which had to be evacuated, but not us.
Sir Wyn Williams: Such was my concentration level, Mr Beer, that I didn’t hear the third noise. So whatever was going on between you and the witness kept it out.
Anyway, we’ll have a 15-minute break.
Mrs Chambers, I don’t expect you to keep yourself in purdah when we have these breaks but just don’t talk about your evidence with anyone, all right?
The Witness: Thank you.
( 11.30 am)
(A short break)
( 11.50 am)
Sir Wyn Williams: Yes, Mr Beer.
Mr Beer: Thank you, sir.
Mrs Chambers you said before the break that when a ticket would come in, you would principally rely on PinICLs or PEAKs that were referenced in a KEL to conduct your investigation.
Anne Chambers: No, you asked me if I would search through the PEAKs –
Mr Beer: Yes.
Anne Chambers: – and I said probably not, you’d link from a KEL.
Mr Beer: Yes.
Anne Chambers: That wouldn’t be how you’d start an investigation.
Mr Beer: No, I wasn’t saying that was the entire range of the data that you would look at.
Anne Chambers: Yeah.
Mr Beer: We looked at the data that you would use before the break but, insofar as you were to look for PinICLs and PEAKs, you would rely on those that were referenced in the KEL?
Anne Chambers: That would be your starting point, if you wanted to – if you needed to look at another PEAK, to –
Mr Beer: So say there were two that were referenced and they were hyperlinked there, would you think, “Right, that’s it”, or would you, on each and every occasion, look at the PinICL and PEAK database to see whether there are any more?
Anne Chambers: Um, no, you – it would depend so much on the individual problem.
Mr Beer: What factors would determine whether you would or would not rely on PinICLs and PEAKs identified in a KEL?
Anne Chambers: Sorry, I’m not thinking this through very well. Um, when you’re investigating a problem that’s come in, you – you’re not necessarily starting by seeing how many times it’s already happened, or whatever. That might then be something that you would do later on in the investigation, but you – so you’re saying, if it’s a known error, a definite known error that has come in, would I then go and look to see how many other occurrences of it there had been?
Mr Beer: Yes, I’m not saying that. I’m asking what your practice was?
Anne Chambers: Yeah, I mean, if it’s a known error and there is a KEL for it already, then it is possible that that should not have come over to third line in any case.
Mr Beer: But we’re necessarily talking here about cases where there is a KEL associated with the ticket that you’re –
Anne Chambers: There is a KEL associated with the ticket but the call has been passed over to us anyway so then we need to look at the circumstances of this individual call and see whether the KEL does relate to it. You know, you do a lot of investigation before you go following all the other links.
Mr Beer: Yes, and I wasn’t looking at the issue of where do you start; I was looking at the entirety of your investigation and, in the entirety of that investigation, the question is: to what extent do you rely on only those PEAKs and PinICLs identified in the KEL as being associated with this issue, or do you look at the PinICL and PEAK database to look for other PinICLs and PEAKs that may be associated with this issue?
Anne Chambers: Yes, in some cases you would.
Mr Beer: What would determine the some cases that you would and those that you wouldn’t?
Anne Chambers: If it looked like it was a repeating problem, that wasn’t – where you needed to get some idea of how often it was happening, then, yes, you would go and look at all the PEAKs and PinICLs.
Mr Beer: How would you know if it was a repeating problem without looking at the PinICLs and PEAKs?
Anne Chambers: Because of our knowledge of the system and the things that we had individually looked at before and whether the KEL said this has happened here, here and here, and what the implications of the problem were. I mean, in some cases, you would – yeah, sorry, I’m finding this rather hard to answer sensibly because it’s not – you know, if you gave me – if …
So if we’re – you’re saying a new problem has been – well, an existing problem is there, we have another call about an existing problem, would I always go and see how many instances there had been? It would depend what – whether it was something that each instance could be dealt with sensibly, individually or whether we felt it was part of a, you know – there was a bigger picture that needed to be identified.
Mr Beer: Okay, I’ll move on but I’ll come back to that later.
You say in your witness statement, it’s paragraph 16, if we just look at it on page 4:
“I am asked whether I consider that the KEL system was adequate for its purpose. Overall, I think the KEL system worked well although there were some problems. For example, many KELs documented similar symptoms, and service tickets could be passed to SSC with the wrong KEL quoted.”
Yes?
Anne Chambers: Yeah.
Mr Beer: When you say “the wrong KEL quoted”, it meant that somebody in the chain before you had identified a KEL that was unrelated to or irrelevant to this problem; is that right?
Anne Chambers: Yes, or it might have looked similar on the surface but they were unable to – they hadn’t realised it didn’t apply, and there might have been a better KEL which they hadn’t found.
Mr Beer: Was that raised as an issue of concern within Fujitsu by the SSC?
Anne Chambers: No, I don’t think so it was up to SSC to improve the KELs so that the right one was found in future. We were the ones who were writing the KELs.
Mr Beer: But you weren’t the one that was doing the associations on a new ticket that was sent to you, were you?
Anne Chambers: No, but if they were – if second line, first line had found the wrong KEL then, you know, we would look at the KELs to see how it could be made clearer in future, so they would – were more likely to pick up the correct one. That was part of our job.
Mr Beer: Was anything therefore done to rectify this problem with the KEL system?
Anne Chambers: Well, it wasn’t a problem with the KEL system it was a problem with the individual – the ways some of the individual KELs were written, if there wasn’t enough information in them for somebody to ascertain between problem A and problem B.
Mr Beer: That’s one way of looking at it: it’s the way that the KEL has been written by the SSC.
Anne Chambers: Mm.
Mr Beer: Another way is that the people doing the assigning in phase 1 and phase 2, first and second line, are just misassociating KELs with the new ticket?
Anne Chambers: Yes, so it is a problem that they have done that and – yeah.
Mr Beer: Was that raised with first and second line support?
Anne Chambers: I’m sure occasionally it was passed back to them that they hadn’t found the right one, but I don’t think it was such a huge – yes, I don’t think it was a huge problem.
Mr Beer: How was it established that the wrong KEL had been quoted on the ticket?
Anne Chambers: Because when I or one of my colleagues looked at the information and the problem, we could see that it wasn’t the right one and that there was a better one.
I mean, we – we wouldn’t have started our investigation only by looking at the KEL that had been pointed out to us. We would have looked at all the evidence available.
Mr Beer: If you had picked up a ticket that had the wrong KEL associated with it, would you go back yourself to the person in first or second line support who had made that association and say, “Look, you’ve associated the wrong KEL here”?
Anne Chambers: Probably not.
Mr Beer: What was the system, therefore, to ensure that first and second line support did not make these mistakes?
Anne Chambers: For me to rewrite the KELs as necessary, so to clarify between the two problems.
Mr Beer: You are again focusing on saying that it’s your fault or the SSC’s fault, rather than people in the first and second line –
Anne Chambers: Yes, yes.
Mr Beer: – making mistakes?
Anne Chambers: Largely, yes. I mean, people do make mistakes you have to base your systems around the fact that people don’t always get it right first time.
Mr Beer: Was there any system of reporting to your manager where you would log “Wrong KEL associated with this ticket”, and he would collect that data up on a monthly, quarterly, yearly basis and then go back to first and second line support?
Anne Chambers: I don’t know. I mean, we might well put a comment on the PEAK saying, “It’s not this KEL; it’s that one”. Whether anybody monitored for that and fed it back, I don’t know.
Mr Beer: If there wasn’t anything in the KEL or the PEAKs or PinICLs to help you, did you have any tools for analysing for the branch concerned a week or a month’s worth of data, or did you need the subpostmaster to narrow the period of the relevant problem down to a reasonably short period of time so you could look at that data line by line?
Anne Chambers: It obviously helped if the postmaster was aware – you know, had some idea of which day or what sort of – are we talking now about balancing problems –
Mr Beer: Yes.
Anne Chambers: – where there’s a discrepancy?
Mr Beer: Yes.
Anne Chambers: Because this was any a very small proportion of the calls we were dealing with. So maybe I’ve been misunderstanding you because I’ve been answering in general terms, whereas maybe you’ve been intending to ask me about specific balancing problems.
Mr Beer: Previously I was asking in general terms about the system of linking KELs to PEAKs and PinICLs.
Anne Chambers: Yeah.
Mr Beer: Now I’m asking about –
Anne Chambers: A specific balancing problem.
The more information that the postmaster could provide, the more – the easier it was, obviously, for us to focus and look at a particular area of concern. And sometimes – I don’t know, we’ll see examples of this, problems with rem in and rem outs. They realised very quickly that something had gone wrong while they were doing that and so then obviously we’d always pull back the complete message store, which contained roughly a month’s transactions. That varied at different times but we’re talking about a month’s transactions.
Mr Beer: Just to stop you there, was that the typical period that you personally would seek data for?
Anne Chambers: That is what was in a counter message store when you retrieved it from the correspondence server, because the data was retained for, I think initially, 42 days and then it dropped down to about 35 days, and so the message store that we got back for a branch would always contain all that data. We would then focus in on any specific areas of problems but, if necessary, we could look over that entire period.
Mr Beer: If a subpostmaster said that they had misbalanced but they couldn’t point out where in the week that had occurred or where in the month later on that had occurred, would you ever refer them back to the NBSC?
Anne Chambers: I would always have a look to see if I could narrow it down to where a problem might have occurred and I can go into some detail as to how I would do that, if you want me to.
Mr Beer: At the moment, would you ever refer them back to the NBSC to provide more detail?
Anne Chambers: If – the NBSC were meant to have taken them through, to question them fairly strongly to see if there were any user errors that might have caused this. If we got a – this type of call, and there was no sign that it had already been through NBSC, then it might well be passed back but we would normally expect the first or second line to have said, “Hang on, you need to go and talk to NBSC first”.
So, by the time it came back through to us, I would almost always – I would have a look anyway just to see what I could see.
Mr Beer: Did you have a methodical process that you applied to each ticket, in terms of steps of investigation that replicated itself time and time again or was it dependent on the nature of the issue identified in the ticket?
Anne Chambers: It would depend very much on the nature of the issue but, you know, getting the message store was always one of the first things for anything counter related.
Mr Beer: What did you do when you obtained the message store? I think this was what you were going to tell me a moment ago.
Anne Chambers: Yeah, you opened it up and it’s this absolutely enormous text file so we used a fairly good text editor that would let us highlight, search things, highlight lines, pull out all the lines that we’d marked. So for a discrepancy call, where we weren’t given any other clues, I would highlight all the product 1 lines – product 1 being cash – pull them out, put them into a spreadsheet which I’d developed a bit, that then – so instead of just a very long, very hard to read text line, it would pull out fields of interest, which obviously would be value, the mode in which this transaction had been done.
I would then do a column with a running total to give you the system cash position at any point in time. So if you say at the start of the week the postmaster has balanced, so he’s declared how much cash he’s got so you have to, you know, at some point assume that that was correct, so you’ve got a starting position, you can then work out your system cash position as you go through by adding on all the cash transactions that have taken place.
Then, at the points at which the postmaster declares cash or declares his overnight cash holding, you can see two other figures – well, at least one. You can see what he or she has declared that they are holding at that point, and if it’s declare cash or an overnight cash holding where they calculated the difference, you can also see what the system calculates the cash to be at that time.
So going through a week or a month, you’ve got all these points where you’ve got two or three figures that you can compare to see how in line they are. Now, if you’ve got a difference between the first – your own running system total and the cash total that the system has calculated at that point, if those are different, then you have a system problem because – of some kind, which you can then investigate and see, well, I think the system cash should have been this but the system is that. Why are they different? What’s not been included? And so there are some of the bugs that are covered which would fall into that category.
And also, if you’re – yeah. I’ll go back to that. But then you’ve got the comparison between what the postmaster has declared he’s got and what the calculated figure is, and that is your discrepancy, which you’re then looking for a cause for.
Now, if you’ve done this over a week, sometimes you can see it’s in step, as it should be, these figures are all in step, except for one day suddenly it jumps and suddenly you’ve got a discrepancy of £2,000. So then, on that day, you look at all the transactions to see if you can see anything, either system error or user error, that could possibly have caused a discrepancy of £2,000.
Mr Beer: Just stopping there, how would you determine whether the discrepancy was user error or system error?
Anne Chambers: You can’t.
Mr Beer: You just said you would determine whether it was system error or user error.
Anne Chambers: Well, you can look. If you can – if you can see something like a rem of the same pouch has gone in three or four times, then that’s fairly likely, either the postmaster has been – got really carried away and has scanned the thing several times, which shouldn’t be allowed to happen anyway, or it’s a good working hypothesis that you have some sort of system error with that. So then you need to look and see exactly what has happened.
But if you look at all these – I mean, you’d start out just by looking at the cash transactions and the different modes. If you can’t see anything anywhere that gives you any sort of a clue, it doesn’t seem to be particularly on one particular day or anything, you may not be able to – in those cases – and it did happen – if there’s no sign of any system error, the calculated system figure is correct, all that is wrong is the difference between the system figure and what the postmaster says – has declared that they’ve got, then, unless you’ve got the knowledge of what has taken place at the branch and have some way of checking that what is recorded on the system actually matches what happened at the branch, then you are not going to get any further.
Mr Beer: We’re going to come back to this a little later today but, in that case, where you couldn’t possibly identify a system error, was the ticket written up as user error?
Anne Chambers: Not normally, no. It would normally be “There’s no evidence of a system error”.
Mr Beer: What was the consequence of writing a ticket up “No evidence of a system error”?
Anne Chambers: It would go back through the lines of support and then it would be up to the postmaster and NBSC to see if they could pursue it any further.
Mr Beer: What do you mean by pursue it any further?
Anne Chambers: Whether – and hindsight is a wonderful thing, but when I first started doing these sort of things, I sort of assumed that perhaps somebody within the Post Office organisation would go and help the postmaster to discover where something might be going wrong.
Mr Beer: Why did you assume that?
Anne Chambers: Because that seemed a reasonable thing to happen.
Mr Beer: Did you have any positive evidence that that did happen?
Anne Chambers: No, and from talking to postmasters when I sort of said “Well, you know, maybe your manager could help”, I didn’t often get any very positive feedback to that suggestion.
Mr Beer: Were you told that in fact what happened was that if you wrote off a ticket or wrote up a ticket which said, “No evidence of system error”, that the consequence of that would be that the postmaster would pay –
Anne Chambers: No.
Mr Beer: – would have to pay?
Anne Chambers: No, I didn’t – certainly early on, I did not realise that.
Mr Beer: After early on, when did you realise it?
Anne Chambers: Um, I suppose when cases started going to court.
Mr Beer: Can you date that?
Anne Chambers: 2005.
Mr Beer: Did that affect the way that you conducted yourself after then?
Anne Chambers: I don’t think so because I still – you know, my job was to try to identify system errors and, you know, you can’t, I think, turn round and say, “Oh, well, it might be a system error but I can’t find it”, not in a case where – not when there’s – you know, there is so much variability, shall we say, on the customer side.
Mr Beer: In any event, we’ll come back to that a little later on. Can we look, please, at FUJ00086462, please. Can we start, please, at page 2.
This is a series of emails that you became involved in, in 2006, concerning the data tree build failure, that we’re going to look at later, but just to orientate yourself, this is some six years into the operation of –
Anne Chambers: I don’t think this is quite to do with that.
Mr Beer: Oh, isn’t it? Well, let’s go down and look at Kimberly Yip’s message at the foot of the page, please. You’ll see you’re not involved in this but it’s the background to it.
Anne Chambers: But I had – yes, yeah.
Mr Beer: You’re not a copy-ee yet.
Anne Chambers: No.
Mr Beer: You’ll see that this is about performance speed, I think.
Anne Chambers: Yes, it was the performance of them producing their balance reports.
Mr Beer: And –
Anne Chambers: But it’s not the same as the data tree problem.
Mr Beer: As the data tree. Okay. Ms Yip sent an email to Graham Welsh. Who was Graham Welsh?
Anne Chambers: Um, customer services manager?
Mr Beer: We’ve got some documents that suggest his job title was Fujitsu’s Strategic Services Manager for the Post Office Account.
Anne Chambers: I think he had various job titles over the years.
Mr Beer: But you would put him down as customer services, essentially?
Anne Chambers: Yes, I think at that point, that was – he was part of the service management team, if not the leader of the service management team.
Mr Beer: Anyway, Ms Yip says to him:
“Please forgive me if you are not the appropriate person to forward this email to.
“I have been contacted again by the POL Service Line to obtain an update on progress on the current Horizon System performance issues.
“One particular branch has been escalated to me [and then identity of the branch is given] and last rollover timings have been sent to me by Anne Chambers, see below:
“From 17.00 the branch started printing the daily report and this continued until [about] 18.30. They then declared stamps and cash, and pressed the Balance report button at 18.37. The Trial Balance was not printed until 21.12 (ie over 2.5 hours later). Much of this time the system was processing the month’s transactions. There’s a gap between … 19.30 and 20.05 where it may have been waiting for input from the [postmaster], but I can’t be certain.
“After the Trial Balance the report was abandoned, presumably because the [postmaster] needed to check and resolve the discrepancies. At 21.27 cash and stamps were redeclared (with some variation from the original), and at 21.28 the Balance report button was pressed again. The second Trial Balance was printed at 22.58 (1.5 hours) and the Final Balance at 23.04.
“I’ve looked at what was going on during the balance report production.
“There was nothing out of the ordinary, apart from the very large number of transactions being processed (about 40,000). The number of transactions processed per second was rather less than we sometimes see, but not significantly so, apart from the period 19.00 to 19.10 when the counter end-of-day processes were running.
“Anne also provided me with some recommendations which I have passed on to the branch and I will ask FS to do a similar exercise to the one above (ie provide timings) when the next TP rollover is completed, 14 June [2006], to see if there are any significant improvements. I have been told about another branch so I am hoping to do a similar exercise. In both cases the rollover times do seem excessive and my worry is that these are not isolated incidents. So in terms of the time it is taking branches to complete the balance process, can FS provide me with details on what constitutes an acceptable length of time, for example, if it takes 4 hours then this is reasonable or if it’s more than 5 hours then it needs investigating, etc. This will then give me a better understanding on what I should be passing on to FS or if I should be passing on the recommendations to implement.
“One of the recommendations was to roll Balance Period every week, can you confirm that this does reduce the overall time taken to roll into a new TP at the end of the period?
“If you need any clarification, [don’t] hesitate to contact me.”
Then if we scroll back up the page, please, we can see that Mr Welsh forwards this to you and to John Burton.
“Anne,
“Can you please comment on the attached …”
Then:
“John,
“This issue is silly in the amount of time and resource being applied for a system that is performing to design … Yes I know but frankly the level of grief and support required is crazy!”
Then if we go to the message at the top of the page, we can see that Mr Burton replies:
“Graham,
“… I see exactly what you mean. By coincidence, I’m reviewing Gareth’s report on this issue tomorrow morning, before it’s submitted to [Post Office]. I gather it quotes some hefty prices for making improvements, but I’ll be better informed after the review.”
Then further up the page, please. Your reply on the same day:
“John,
“I’ve looked at many branches now, and they range from very slow to horrifically slow when rolling over stock units. It does vary depending on the particular process followed at each branch, and if you break it down into various components each may appear to be (just) within 4 [hours] as long as the weekly rollover used to be, but the impact on the postmasters is horrible.
“There have been some piecemeal changes to try to improve certain areas, but most if these have made little improvement, and overall, may have been a waste of effort.
“As I see it, there are two main problems:
“1. The balancing process repeatedly scans and rebuilds the data tree. This was identified as a problem at least 6 months ago, and improvements to this are, I think, what Gareth is proposing.
“2. Counters are inadequate for the applications now being run on them and do run generally slowly at times. This hasn’t really been fully investigated, and is really difficult to quantify or prove that it is happening – the only evidence is what the [postmaster] reports. It is however adding to the customer dissatisfaction and could only get worse even if we improve balancing.
“I am not at all happy about fobbing postmasters off and telling them that the system is working as designed when it is plainly inadequate for the job. I am also very unhappy that it has taken six months even to get to the point of starting to consider whether [Post Office] will pay for improvements.
“I too would like guidance on when 2nd and 3rd line support should investigate further. Our current response has to be ‘yes, we know balancing is very slow, it is being investigated’ – what else can we say?”
You said in the course of your reply there “they”, that’s the times, range from very slow to horrifically slow. How wide a sample did you take when giving that answer, if you can recall?
Anne Chambers: Um, hundreds, I think. I can’t exactly remember but I did – we got calls coming in about it. I haven’t recently seen any call that I sent off to development but I’m sure we did, and the initial response that we got back from development was “Oh, well, it was agreed with Post Office that, um, it wouldn’t take more than four times as long as” – because they used to have to roll over every week. So now they’re only having to roll over once every four weeks, although they can still roll into a new balance period each week, and apparently it had been agreed that, as long as the overall process was no longer than four times what it had been previously, that would be all right.
But, in practice, it was having a big impact on branches, which I was well aware of.
Mr Beer: What kind of delays are we therefore talking about?
Anne Chambers: Well, you had some of the timings below, so, you know, it was –
Mr Beer: Were they typical, the timings we saw below or atypical?
Anne Chambers: That was not untypical.
Mr Beer: You said the impact on postmasters is “horrible”. What was the horrible impact on postmasters?
Anne Chambers: They were having to – I’m sure the postmasters could answer this one for you but they were having to sit there after end of trading on Wednesday and, instead of getting it all done and being out of the door in an hour/an hour and a half, or whatever, you know, they might still be there five, six hours later.
Mr Beer: With a final balance, as we saw in that example, of just after 11.00 pm?
Anne Chambers: Mm, and obviously, if during the process, they did they’d got discrepancies, which are not unusual things to happen at branches, but then they would have to go back and check and perhaps recount their cash and look for anything. And so, you know, it wasn’t a sit down, press a button and off it all goes. They were having to do a great deal behind the scenes.
Mr Beer: You say the problem had been identified six months ago but nothing effective had been done about it?
Anne Chambers: I think we’d been aware of the problem since the switch from every week cash account periods to where they changed to balance periods and trading periods.
Mr Beer: Wasn’t that in 2004?
Anne Chambers: Um, I thought it was later than that. Maybe – I’m not sure. I can’t remember.
Mr Beer: But, in any event, this email suggests that you knew that the problem had existed for at least six months. What had been done in that six months, to your knowledge?
Anne Chambers: I don’t know.
Mr Beer: You say, “I’m not at all happy about fobbing off subpostmasters and telling them the system is working to design”?
Anne Chambers: Yes, I wasn’t because I had spoken to quite a few of these postmasters and I could tell how unhappy they were. I think this email – I obviously was trying to get my point across forcefully and I was slightly sticking my neck out, but I felt I was a little bit closer to the people who were having the problems, perhaps, than someone like John Burton, who I think was the counter development manager.
Mr Beer: Is what you had been doing for at least that six-month period then been to fob off subpostmasters?
Anne Chambers: No, I would have been answering the calls and trying to explain that it was expected that it would be a slower process now that they were having to do, um – now that the process had changed. But I did feel it – I was concerned that it was having a big impact and that, as far as I had seen, nothing very much had changed, although I think there were a couple of code fixes that this suggests that something had been changed, that was meant to make it better but perhaps didn’t really help very much.
Mr Beer: Mr Welsh had said that the issue was a silly one and that resources within Fujitsu were being applied to a system that was performing to design. You were unhappy about telling postmasters that the system was performing to design, correct?
Anne Chambers: Yes, given the impact that it was having on them.
Mr Beer: The suggestion that you should say to subpostmasters the system is performing to design, was that indicative of a more general approach that you were required to take within the SSC, ie “Don’t reveal the true position that we know about publicly or to the subpostmasters, just say that the system is working well and to design”?
Anne Chambers: I said it in this case because that was what I had been assured of but, no, I would not have said it in other cases where they’d had a problem that was caused by a system error. In that case, I would say to them, um, “Sorry, the system has – there’s an error here, this shouldn’t have happened. It’s a fault in the system which we’ll be investigating”.
Mr Beer: Was there pressure on you in your communications with subpostmasters not to reveal errors in the system?
Anne Chambers: No.
Mr Beer: On this instance, on this email that we’ve got, it tends to suggest that there had been a message that you were required to deliver. You had been fobbing them off and said the system is working to design.
Anne Chambers: Um, no, I think I would have said, “I know it’s awful but I am told that the system is working to design, but we are still looking at it”, Something like that. I wouldn’t have tried to pretend that it wasn’t a problem.
Mr Beer: Scrolling up the page, please. We can see Mr Burton’s response:
“Anne, Graham,
“I reviewed Gareth’s feasibility report and costings this morning, so understand things better than I did. His report is based on a great deal of prototyping work that has been done over the last few months – of the order of 100 man days. That work looked at a number of options, and has homed in on the one that gave the best improvements – along the lines you mention in your first point.
“The report should go into [Post Office] next week. It’ll then be up to them whether or not they want to pay us to do the work. If they decide to go ahead, we’re looking at a likely delivery date of first calendar quarter in 2007. That would give around 2 years of useful life before being overtaken by HNG-X.
“I understand your frustration at having to deal with irate postmasters and having to tell them that the system is working to its spec. We can only hope that POL do agree to funding this work, so that you then have something positive to say.
“I can’t see much point 2nd and 3rd line support doing further investigation, when we now know what needs to be done to make a substantial improvement. Please say, Gareth, if you disagree.”
I should say that Mr Jenkins was copied into that email. Were you content with that response?
Anne Chambers: Yes, I think so, I think it looked, you know, at least something was happening.
Mr Beer: To your knowledge, did the Post Office ever pay for the improvements that were proposed or did they instead wait until Horizon Online was rolled out?
Anne Chambers: No, something did change and it did improve.
Mr Beer: When was that?
Anne Chambers: I can’t remember.
Mr Beer: Can you remember the nature of the improvement?
Anne Chambers: No. I almost certainly would have looked to see – you know, to make sure that it really had made a significant difference.
Mr Beer: Sorry, can you say that last answer again? I was distracted.
Anne Chambers: I am sure that when the change did go in, and I can’t remember when that was, I would have had a look to see if it had improved the time it was taking for some of these worst-affected postmasters.
Mr Beer: Can we look at a different issue but on the same topic of improvements in the system, and look at POL00001265.
You’ll see that this is a PEAK dated – you’ll see the opening line of the PEAK under “Progress Narrative” of 27 March 2006.
You’ll see under the summary two lines above, what the summary of it is, namely, a “Harvester Exception”. Can you explain in brief terms what a harvester exception is, please?
Anne Chambers: Right. So the transactions are written to the message store on the counter from where they replicate to the message stores at the correspondence servers. Overnight, processes run to harvest transactions so they can be sent on to various sources. So, for example, all the bill payments would have to go off to the various companies whose bills are being paid; all the EPOSS transactions were harvested along with others to go to Post Office.
In this case, for some reason it’s trying to harvest a message for EPOSS, and I can only see this top bit so far, but the message written on the counter presumably does not have the mode field which should have been included in it.
Mr Beer: Because it was blank?
Anne Chambers: That appears to be the case from all I can see so far.
Mr Beer: Yes, and then if we scroll down, please, you make an entry at the foot of the page on the same day. The ticket having been assigned to you. Your entry of 27 March at 16.12.36:
“I have repaired the problem transaction and will check tomorrow that it has been sent okay.
“As far as I can tell, no call has gone to development about this. To summarise,
“Some messages get written with a null Mode attribute. The root cause of this has never been resolved.
“Changes have been made to the harvester agents so that the messages with [then you put a character string] can be [installed] when Mode is missing …
“MailsBalance messages have [and then you put some more character strings]. This was spotted soon after their introduction in January, and I did intend to raise a PEAK, but don’t seem to have done so. At the time it was thought to be benign.
“MailsBalance messages with missing Mode are now causing number of missing harvester exceptions (5 on the reports for 24/3).”
What does that mean, for 24 March?
Anne Chambers: 24 March, yes.
Mr Beer: “Each has to be repaired individually.
“So we need to sort out the [character string] issue. This could be fixed at either the agent, or in the Mails scripts. If it can be fixed fairly soon in the scripts, I think that will be the better option rather than making the agent cope with what is basically a typo.
“There are example messages in the attached reports, or I can provide a messagestore if required. Routing to Mails [development team, essentially].”
Yes?
Anne Chambers: Mm-hm.
Mr Beer: Then in the next messages, if we scroll down, I’m not going to go through them in the interests of time, no need to read them, we can see that your suggestion about the investigation of the root cause was not taken up and a decision was taken not to do that because it wouldn’t be cost effective, given the limited shelf life of the Horizon counter application?
Anne Chambers: Mm-hm.
Mr Beer: Can we see your response to that on page 3 of the document, please. At 13.54.33, that’s it, third up, “Response noted”. You say:
“I never really expected the root cause to be investigated or fixed. The typo which caused the agent circumvention to fail was fixed a long time ago. Closing call.”
Anne Chambers: Mm-hm.
Mr Beer: You had earlier suggested that the root cause be investigated and fixed.
Anne Chambers: No, I think I had said just change where it was application mails, just make sure –
Mr Beer: Do you want to just go back up to that at page 1 the last few lines?
Anne Chambers: Yeah, sorry.
Mr Beer: Page 1, scroll down. Thank you.
Anne Chambers: No, I said we need to sort out the applications mails issue. That wasn’t the root cause. The root cause was the null mode attribute.
Mr Beer: So the message you wrote at the end “I never really expected the root cause to be investigated or fixed” –
Anne Chambers: Yeah, that’s the messages with the null mode attribute.
Mr Beer: I see. So, in essence, you’re saying here that what you expected to be done was done; is that right?
Anne Chambers: Yes, yeah.
Mr Beer: Thank you. That can come down.
In your witness statement, paragraph 17, there’s no need to turn it up, you say that the SSC and fourth line support development did not always know how many branches had reported a particular problem because the tickets reporting that problem hadn’t been sent through to the SSC. Yes?
Anne Chambers: Yeah.
Mr Beer: Was that a problem?
Anne Chambers: Um, it was if we didn’t therefore have an idea of the scale of a problem or how many branches were being affected.
Mr Beer: Was that known within Fujitsu, that there was a problem in the design of the system as a whole, in that relevant information was not being passed up to the SSC?
Anne Chambers: I don’t know. Um …
Mr Beer: Well, you’ve identified this as a problem.
Anne Chambers: Mm.
Mr Beer: Was that problem discussed amongst your team, raised with your manager and then escalated within Fujitsu and then within the Post Office?
Anne Chambers: Not to my knowledge.
Mr Beer: Why not?
Anne Chambers: But it –
Mr Beer: If this was a problem in the system that meant that the scale of any identified defect was not known, why wasn’t that addressed?
Anne Chambers: I don’t know.
Mr Beer: It would be relevant –
Anne Chambers: There would be other ways of finding out that information. I mean, it would depend very much on the type of problem but we’re not talking here about things that would have a – well, we shouldn’t be talking about things that would have a significant effect on individual branches.
Mr Beer: How would you –
Anne Chambers: But I think it was just understood that, you know, this was the process, was that first and second line are meant to filter out known errors. That is why they are there. If you’re saying “Well, first and second line need to send everything through anyway”, then you can almost say what’s the point of having them?
Mr Beer: Would you agree, though, that with the benefit of hindsight, this is a problem or a defect within the system?
Anne Chambers: I think there are – there were certainly some areas where it would have been a lot better if SSC or somebody had had more of an insight into how particular problems were affecting the entire estate.
Mr Beer: I mean, that’s a key issue, is it not, for both the Post Office and Fujitsu: a person has identified a problem, to what extent has that problem in the past afflicted the estate or to what extent is it currently afflicting the Post Office estate?
Anne Chambers: Yes, but there are other ways of finding out that information besides having calls passed through for every time it’s reported. I mean you could argue that then you’re dependent on the postmaster actually noticing and reporting the problem and we’d also have other means of seeing how often a specific problem had occurred by, for example, checking the events for the whole estate over a certain period of time. It would just depend what the signature of the problem was. In many cases, we would be able to see where else had been affected.
Mr Beer: Was that habitually done, that when a problem – when a ticket came in, you would adopt that approach of checking the message store for the entirety of the estate?
Anne Chambers: Not when a ticket came in but when an underlying problem had been identified, then certainly, later on in the life of Horizon and, in particular, HNG-X, yes, that was done very rigorously. But I cannot say that that was done the whole time –
Mr Beer: When you say later on – I’m sorry I spoke over you. When you say “later on”, what do you mean?
Anne Chambers: Certainly from the introduction of HNG-X, I think we got a lot better at tracking down every instance of problems without the postmaster having to report it to us.
Mr Beer: Why wasn’t this done before the introduction of Horizon Online in 2010?
Anne Chambers: I think in some cases it was done but perhaps not so rigorously and I don’t know why not.
Mr Beer: Well, can you help, please? What determined whether before 2010 you would say, “Well, I’ve got a problem here, it’s on this ticket. I need to make a decision on whether to check the extent to which this problem is afflicting other parts of the estate”?
Anne Chambers: It would, again, partly depend on what type of a problem it was, how easy it was to do the sort of checks. Um –
Mr Beer: Were there any rules on this? Anything written down?
Anne Chambers: No, I don’t believe so.
Mr Beer: Was it down to individual discretion of the 25 of you?
Anne Chambers: Yes, or probably more down to that. But, I mean, a problem of any scale, it’s unlikely that it’s just one person ends up looking at it anyway, so you wouldn’t – it wouldn’t be sort of one in 25. But – and it would depend on obviously on what sort of a problem was and what sort of impact it was having.
Mr Beer: What do you mean by what sort of impact –
Anne Chambers: Well, if it was affecting branch accounts in some way.
Mr Beer: If it was affecting branch accounts in some way –
Anne Chambers: Then –
Mr Beer: – then would you habitually do –
Anne Chambers: Then you would –
Mr Beer: Hold on. Would you do an estate-wide check to see whether other branches’ accounts had been afflicted by the problem identified?
Anne Chambers: You might well attempt to.
Mr Beer: What would determine whether you might well?
Anne Chambers: Whether there was some clear-cut way of identifying these branches that –
Mr Beer: Was there a clear-cut way of identifying –
Anne Chambers: It would depend on the problem. You can’t generalise.
Mr Beer: Would you escalate issues where you think this is a significant issue that might afflict other branches or some other branches?
Anne Chambers: Again, I think that was something we – that happened more subsequently – with HNG-X onwards.
Mr Beer: You addressed this, if we can look at it, please, in paragraph 53 of your witness statement, which is on page 17. You say in the first sentence:
“I am asked whether Fujitsu took proactive steps to identify bugs and/or discrepancies in branch accounts caused by the same.”
Then reading on four or five lines you say:
“If a bug was found to be affecting branch accounts which had not caused a reconciliation report entry, we would do our best to identify all branches affected, as we did for Bug 3. However, I cannot say that this was done consistently for all bugs ever found, especially in the early days of the project.”
So my question is why was it not done consistently for all bugs?
Anne Chambers: Yes, I think perhaps we – I don’t know. I can’t answer that question.
Mr Beer: To what extent was there liaison with the Post Office when identifying whether there should be an estate-wide search for the extent of a problem?
Anne Chambers: Um, again, I know that a lot of the problems that happened, HNG-X onwards, there was liaison with them and discussion as to how to do these searches and who should be undertaking them.
Mr Beer: What about the decade before then?
Anne Chambers: Um, I think one reason I’m finding it so hard to remember is that I’ve seen very little – very few PEAKs and things from that period, sort of 2007 to 2010 in particular. So I – really, I’m finding it very, very hard to remember, you know. I can’t turn around and say, “Oh, yes, we did this, this and this”, because my memory hasn’t really been jogged by specific instances.
Mr Beer: How did you interact with the MSU in this situation?
Anne Chambers: Er –
Mr Beer: Did you highlight this kind of situation to them so that they could let all branches know to be on the lookout for a known problem?
Anne Chambers: Um, that wasn’t really MSU’s role. That was very much a branch-by-branch basis.
Mr Beer: Was there any liaison with the Post Office about – I’m talking about pre-2010, here – notification of the discovery of a bug and to look out for symptoms or signs of its existence?
Anne Chambers: I’m sure there were some but I’ve not seen any specific examples in the last six months.
Mr Beer: You say in paragraph 212 of your witness statement, if we go to that, please, it’s right at the end on page 63:
“A point of frustration with the system, was that the users, namely the subpostmasters, were not our clients and there was a practically limit as to the extent to which we could work together with them to investigate problems.”
Are you, by this, suggesting each problem that was called in by a subpostmaster was treated separately and that as a result, there wasn’t any oversight of any wider system issues?
Anne Chambers: No, I think what I was trying to say there was that, you know, there were cases where you just wished you had some way of knowing what had happened at the branch, and could – and there was some way of getting some more information.
Mr Beer: Did you, in the first ten years of the project, take the view that tickets were sent in, were addressed piecemeal, one by one, problems were patched up, and there was no analysis of whether the issues the fundamental issues raised by the tickets needed to be addressed coherently through any redevelopment or redesign?
Anne Chambers: Um, no, I think there was – there certainly was analysis done and, where we saw patterns emerging, we did try to make sure those problems were progressed.
Mr Beer: Can we look at an example, please, of where you suggested, I think, that others take a wider look at the system. FUJ00086490, thank you. If we expand it. We’re going to come back to this PEAK in due course when we look at Bug 10, the data tree field – sorry, the data tree build failure. I just want to look at it now to explore couple of issues with you, just on the back of this PEAK.
You’ll see that the summary of the problem is discrepancies when declaring euros or cash. Can you see that? It’s the sixth line in. Thank you.
We can see, if we scroll down a little bit that it was raised on 18 May 2007, under the first entry on the progress narrative. Then if we can look at page 3 of this PEAK, please, and the entry at 10.45.06 on 24 May made by you, and you say:
“This branch reports they have been having problems since March 2006 …”
Remembering we’re now in May 2007:
“… where they do declarations, do further transactions (usually transfers or rems), redeclare and then get a discrepancy equal to the value of the recent transactions.
“Two recent examples (all times UTC).
“Wednesday 16th May: 2000 euros transferred out at 14.23 then reported as a discrepancy during trial balance at 15.54.
“Monday 21st May: 8 cash transactions totalling £2,465 between 9.35 and 10.16, ignored when cash declared subsequently at 10.20.
“The problems are always in stock unit MS, which is an individual stock unit (hence a variance check is run automatically after a cash declaration). Almost always using the gateway for this stock unit.
“These types of errors are not that unusual these days (I would say much less rare than a couple of years ago?) We usually quote [then you give a KEL] – I think the outcome of those investigations was that the Riposte message port was possibly sometimes failing.
“However, given that this problem is not uncommon at this branch, and also that [another PEAK] found a problem within Horizon that gave very similar consequences, I think this might be worth a recheck. I appreciate that with [Horizon Online] coming up, it may not be worth pursuing this, but this problem is causing a number of postmasters to have serious doubts about the reliability of their systems since they are very displaying incorrect figures.”
If we also look at your entry on page 4 at 11.16.12, just at the foot of the page there, thank you. So we’re three months on now. Sorry, two months on now.
“This problem is continuing to affect a number of branches, especially larger ones where they have individual stock units and commonly where they have transferred out cash (which then appears as if it is still in the stock unit – hence they think there is something wrong with the transfer mechanism). The [postmasters] are not all happy that the system is giving them incorrect figures, causing much confusion and potentially creating opportunities for fraud.”
So in this PEAK, we see how a multi-branch issue involving discrepancies was suggested to be brought together for a proper fix by you; is that right?
Anne Chambers: Yes.
Mr Beer: But would this be right: that was as a result of your essentially anecdotal experience of an increasing number of calls of a similar nature?
Anne Chambers: Yes, I was probably keeping an eye out for these type of calls.
Mr Beer: You kept an eye out –
Anne Chambers: Mm.
Mr Beer: – for these calls, and some branches had experienced a continuing problem for a number of years and various workarounds were employed?
Anne Chambers: Yes.
Mr Beer: But, ultimately, the suggestion of a fix was essentially ad hoc by you, wasn’t it?
Anne Chambers: I’m not sure what you mean.
Mr Beer: I mean that it was just you keeping a weather eye out for whether this was a problem that was affecting consistently a number of branches rather than the system that Fujitsu employed itself detecting whether this was a consistent and ongoing problem?
Anne Chambers: Yes.
Mr Beer: Was that essentially the approach adopted in the SSC? It was down to you, as an individual, to keep a weather eye out to see whether there was a big problem or not?
Anne Chambers: Yes, I think that’s probably true.
Mr Beer: Would you accept that that rather patchwork approach, dependent on 25 individuals not each knowing what the other is doing, not each knowing what the other is seeing, may leave frailties in the system?
Anne Chambers: It could do, although I would say – I mean, I wasn’t just looking at the calls that I had dealt with. I would have been – have noticed that some of my colleagues had also had similar calls, and in fact we would have discussed it together. But the process that we were meant to follow was to, you know, you passed the call over to Development, and then it is their responsibility to produce a fix at some point in the future.
I mean, there were teams, or there were people who would see which calls were with Development sort of potentially in the pipeline and so on. But yes, them actually having – anybody having the knowledge of something like this, where it was a continuing, potentially serious, problem, that information was not necessarily being fed in.
Mr Beer: Thank you very much, Mrs Chambers.
If that’s an appropriate moment for you, sir, I would suggest that we break until 2.00.
Sir Wyn Williams: Well, to give everybody their full hour, 2.05, Mr Beer, provided the time on my machine in front of me is accurate at 13.03.
Mr Beer: I’ve got 12.59.
Sir Wyn Williams: Really? Oh. It just shows how reliable this system is. Fine. Let’s compromise: 2.02.
Mr Beer: Thank you very much, sir.
(1.00 pm)
(The Short Adjournment)
(2.00 pm)
Mr Beer: Thank you, sir. Mrs Chambers, can we continue to examine in general terms how tickets were processed in the SSC focusing in particular on the extent to which it was a part of the system to identify the extent to which an error, bug or a defect afflicted other parts of the estate.
Can we start by looking, please, at FUJ00082274. This is a KEL titled “Multiple cash declarations may cause incorrect figures in Discrepancy, Variance and Balance Reports”. It was raised, you’ll see, by Mark Scardifield on 15 July 2005, and was last updated by you on 27 November 2007.
If we look at the symptoms, please, thank you:
“A cash declaration was made in ‘Stock Balancing’ for the amount displayed on the Snapshot. When the Cash Variance was checked afterwards a Gain of £45.05 was displayed [then a string of characters]. May get [postmasters] calling in to state that they’ve been declaring cash but they have been getting varying discrepancies reported even though they’ve been declaring the same amount of cash each time. Or that they have done a transfer but are then getting a discrepancy equal to the amount of the transfer, or that the system hasn’t transferred the cash out of the stock unit.”
In terms of the “Problem”:
“The underlying problem is that we cache the current trading position for a Stock Unit and rely on a mechanism (in Riposte) to notify us of new transactions across the outlet to keep this cache up to date. When this fails it affects Discrepancy, Variance and Balance Reports and has the effect of presenting the clerk with the incorrect information. This will be potentially confusing and may lead to the clerk making unnecessary corrections. These will in turn show up as future inconsistencies (eg nothing gets lost in the end).”
Is this right, that this is identifying another problem with the Riposte system?
Anne Chambers: Yes, I think the underlying error was thought to be in Riposte itself.
Mr Beer: Is it identifying a known problem with Riposte?
Anne Chambers: It’s identifying a problem in Riposte. I’m not – it’s known, in that it’s now been documented.
Mr Beer: I see. So it’s recording that some figures might be wrong, which might lead a subpostmaster or a clerk to make updating changes, which would then create a problem when the figures were corrected?
Anne Chambers: If they did – if they were to make those changes, then yes, they would then have to undo those extra changes.
Mr Beer: So in order to ensure no losses, if the postmaster made the unnecessary corrections, would they need to recognise the error and reconcile it, whether by making a transaction correction or getting a transaction correction, or otherwise?
Anne Chambers: Yes, it would depend at what point they noticed that something was wrong. In the cases that I can remember seeing, including the call that we looked at this morning, the postmaster was very aware that these problems – that the figures were wrong and phoned in to complain and was advised to reboot, and so on, to avoid it. So, in that case, he or she didn’t take any extra action, except doing a reboot and then when they started again the figures were all correct.
Mr Beer: The solution is given as:
“The Declare Cash problem clears itself overnight. If the [postmaster] logs a call on the day as having problems, ask him to try the following workaround. The clerk should log out of the affected counter. Another clerk attached to a different (individual, not shared) stock unit should log into the same counter, declare cash for his own stock unit, then log out. The first clerk can now log in to the same counter and declare cash again. The variance should be correctly recalculated. Alternatively log on to a different counter and do the cash declaration there. If the workaround is not successful or the problem does not clear itself overnight, send a call to the SSC, otherwise no call is needed. November 2007: a fix is being piloted and is likely to be sent to the whole estate in January.”
Presumably that will be January 2008?
Anne Chambers: Yeah.
Mr Beer: So that fix appears to be planned to be sent out in January 2008 for a problem that was identified in July 2005?
Anne Chambers: I think the problem in 2005, the one that was found during testing, it’s not – I can’t tell if it was exactly the same problem that then persisted. I don’t recall – well, from what I’ve seen on the various bits of information that – documents that I’ve reviewed about this problem, it looks as if it started happening in Live, or at least was noticed more in Live, sometime in 2006.
Mr Beer: But still a year and a half, two years?
Anne Chambers: It’s still a very long time, yeah.
Mr Beer: Can you explain was that normal, it would take a year and a half, two years, for something to work its way through the system for a fix?
Anne Chambers: That was unusual, it depends so much on the sort of what problem it is. I mean, if it’s a very contained problem, easy to reproduce, “We know if we do ABC, this is going to happen”, then those sort of problems are usually relatively easy to fix and to test and to get the fix out into Live. This was not such a straightforward problem. I’m not sure if anybody ever actually managed to replicate it. We didn’t have a sequence of events that would make it happen, and so, in the end, the fix, I believe, was quite a large one, which involved sort of redesigning and rewriting an area of code to sort of completely change the way it worked, and so, obviously, that’s both a bigger job to do, and then it has to go through a full test cycle and because it’s – something like that could have – is more likely to have knock-on effects than something which is very self-contained.
Mr Beer: At the foot of the page under “Evidence” the KEL says:
“If there is any pattern of failure, eg it is occurring on the same counter repeatedly, can the audit log … be recovered of this occurs please?”
Anne Chambers: That’s obviously the counter log that I had forgotten about. I had a feeling that there might have been another one and that must be it.
Mr Beer: Would that log not routinely be obtained where there was a concern over a system failure raised like this?
Anne Chambers: Yes, it probably was, because I think that’s probably the one that had some diagnostics and things written to it.
Mr Beer: That’s what I was going to ask. What is the audit log being referred to there?
Anne Chambers: As I said, I think that must be the file, and there was – probably looking at that, I’m not sure if that was a day of the week, so there would only have been seven of them or if that would have been a date in there, but it wouldn’t be something that persisted for very long.
Mr Beer: Why would the audit log only be of interest in if there was a repeated failure?
Anne Chambers: Um, I think it’s because then – no, it probably would have been of interest anyway.
Mr Beer: Why does it say, if there’s a pattern of failure –
Anne Chambers: I don’t know. I don’t know who put that on there.
Mr Beer: – obtain the audit book?
Can you think of a reason why one would only obtain the audit log if there was a pattern of failure?
Anne Chambers: No. I think you’d probably obtain it anyway but, as I said, I’m afraid until this moment I had totally forgotten its existence.
Mr Beer: Can we look at a PEAK, please, POL00028746. This is a PEAK 0103864 raised, you’ll see under the “Progress Narrative”, on 3 June 2004, after a subpostmaster reported problems with transfers. You can see that four lines in:
“Summary: [postmaster] reports that he had a problem with some transfers.”
Yes? Can you see that?
Anne Chambers: Yes, sorry.
Mr Beer: Thank you. Was the problem on this, recorded by this PEAK, a discrepancy arising after transfers which were being replicated twice?
Anne Chambers: Yes, they – you should only – this – a transfer is when you’re moving cash or stock from one stock unit to another, and you should only have been allowed, for one transfer out, just to do the transfer in the one time. If, for some reason – and there were various underlying reasons why it might be possible – you could do the same, accept it in twice, then you would have a discrepancy and also a receipts and payments mismatch.
Mr Beer: If we look at the foot of the page, please, at the entry for the 3 June at 11.25, about 10, 12 lines up from the bottom, thank you:
“Information: [postmaster] reports that the transaction log shows only one transfer out for each item but the transfers in show so that each transaction has been accepted into the BB …”
Anne Chambers: That would be the name of the stock unit.
Mr Beer: “… twice and this has caused him a discrepancy and he would like this investigated. This call was passed to HSH from Tier two at the NBSC and they have also requested that the problem be investigated.”
Then if we see what happened, if we go over the page, please, and look at the foot of the page at – if we go down a bit please. 10.19.14 on 8 June, passed, I think, to your colleague Catherine Obeng, yes?
Anne Chambers: Mm-hm.
Mr Beer: “All nodes were connected at the time that the transfer in [transactions] were attempted.
“No session transfers took place that day.
“Event log from node 4 suggests that Riposte replication had not been successful and so while node 3 had successfully TI the [transactions], this information was not apparent to node 4 thus it was perceived by node 4 that those [transactions] were outstanding waiting to be TI. Therefore when the user … was logged on to node 4, he was presented with Outstanding Transfer message which had to be accepted or declined. The user chose to accept them even though he tells me that at this stage he was a little concerned because he was certain that [the other user] had already transactioned on node 3. This has created a discrepancy on their Cash Account of £22,290.00. Also the host has reported a reconciliation error … for £44,580.00.”
Anne Chambers: Yeah.
Mr Beer: So, essentially, reading this, was this a software error which had then been compounded by an action taken by the subpostmaster in the face of an apparent error?
Anne Chambers: Um, that looks like the likely reason.
Mr Beer: A significant sum of money was involved –
Anne Chambers: Yes.
Mr Beer: – by way of discrepancy?
Anne Chambers: Yes, and it did show up on one of our automatic reconciliation reports so, even if the postmaster hadn’t realised what had happened, we would have been investigating it anyway.
Mr Beer: How complete was the coverage of that automated system?
Anne Chambers: Certainly anything that showed up – I mean, yes, all entries on the major reconciliation reports would have calls raising for them and SSC would have looked at each one.
Mr Beer: We can see if we go to page 4, please, that two-thirds of the way down, at an entry for 15.11.29, Martin McConnell says:
“Attached will be a spreadsheet and a set of transactions that I think are responsible for the error. The spreadsheet is ALL transactions that account to Stock Unit BB and I have presented at data view for the discrepancies committed. I do not even know if I am expected to be doing a summary inspection nevertheless the attached give a ‘view’ as stated.
“As far as the system is concerned, there seems to be a flaw in Riposte informing Counter 4 that the transfer object [a number of stock unit BB is given] has been raised. This is an Escher Riposte problem as far as I am concerned, the transfer mechanism already has a belt and basis approach. If we can’t trust the underlying software to replicate this information through to all other counters, what are we supposed to be able to do? If someone wants to press for a fix, I suggest this is pointed at the Escher team for them to sort out.”
The question there – and I realise this isn’t you writing it – “if someone wants to press for a fix”, why might that even be an issue, whether you would press for a fix or not? A substantial sum of money, by way of discrepancy here, drawn to the attention of Fujitsu by the subpostmasters themselves, what would be the reasons for not pressing for a fix?
Anne Chambers: Whether it could be worked out in precisely what circumstances this error had occurred, I suppose. I mean, these transfers were happening successfully at many branches at many times, so I agree there’s investigation needed to try to find out why this particular one didn’t work. Could we see if Catherine said anything else before she passed it over to development? Let’s go up a little bit.
Mr Beer: Let’s go up to page 3. Then scroll down.
Anne Chambers: Right, so maybe there isn’t. I’m sorry, I can’t see the whole picture.
Mr Beer: If we zoom out, please, then it becomes too small. If you zoom in and just go to the entry starting 8 June at 14.45.45, thank you.
Anne Chambers: I just wondered if there had been any other information, any events or anything that gave any more of a clue as to why the problem had occurred, but I can’t see anything else that’s been recorded on there. So yes, I mean, obviously, you’ll think you want this problem fixed but, if you can’t reproduce the problem or get any closer to the circumstances which has caused it, it’s not necessarily going to be fixable.
Mr Beer: So what, it’s an error that just stays on the system?
Anne Chambers: If it’s something that has happened this one time and we cannot see any reason for it, then possibly, yes.
Mr Beer: What about if it was for a smaller sum of money, for £100, and the subpostmaster was told before it got to the SSC, “You just need to make that up, that £100”?
Anne Chambers: No, anything like that, because – as well as it being reported by the branch, it would also have been on the reconciliation reports, and we looked at those regardless of the amount of money –
Mr Beer: There was –
Anne Chambers: – because it is –
Mr Beer: I’m sorry.
Anne Chambers: No, because it is indicating, yes, there is a fault in the system.
Mr Beer: Can we go on, please, to page 5, and look at the second, third entry on the page, also from your colleague Catherine Obeng at 16.30.17:
“Martin McConnell’s recommendation is to put the £22,290.00 which is adrift into the Suspense Account.”
Was money adrift at this point? Is that the right way to describe it?
Anne Chambers: Well, it’s a loss that the branch should not have been liable for.
Mr Beer: “I am routing call [says Ms Obeng] to MSU to raise an Error Notice to inform POL of this incident. Also please notify NBSC to contact the [postmaster] and advise him of this course of action.”
So we’re now, I think, on 11 June, the incident having arisen on 3 June.
Anne Chambers: Yeah.
Mr Beer: So somebody was to contact the NBSC – the MSU were?
Anne Chambers: Um, the first or second line support would probably be expected to contact NBSC.
Mr Beer: Then NBSC have got to contact the subpostmaster themselves?
Anne Chambers: That’s what it says here, yes.
Mr Beer: How would NBSC communicate accurately what had been established on the previous five pages of this document?
Anne Chambers: I don’t know. It’s not made terribly clear there that, you know, this was system error, but that we had informed Post Office and that the branch should expect an error notice in – at some point. There is an error there in that it says calls routed to MSU to raise an error notice. MSU couldn’t do that, they would raise a BIMS report, which would be sent to the Post Office financial people and then they could raise the error notice.
Mr Beer: “BIMS” being a Business Incident Management report?
Anne Chambers: Could well be.
Mr Beer: That was the acronym used, I think, at the time?
Anne Chambers: Right, I couldn’t remember what it stood for.
Mr Beer: Okay. So here we are at 11 June with the SSC telling other parts of Fujitsu to tell the Post Office to contact the subpostmaster to inform him of this incident. Did the NBSC have access to this PEAK?
Anne Chambers: The postmaster already knew about the incident because he had raised the call in the first place.
Mr Beer: Yes. Sorry, to inform POL of this incident:
“… please notify the NBSC to contact the [postmaster] and advise him of this course of action.”
Did the NBSC have access to this PEAK so that they could say to the subpostmaster “Don’t worry, it’s not you, there’s a bug”?
Anne Chambers: I don’t know, and I don’t know what the second or first line support, who would have notified the NBSC, would have passed through to the postmaster.
Mr Beer: Then if we scroll down, please, to 22 June, 15.36.46:
“The Call record has been transferred to the team: MSU …”
Yes?
Anne Chambers: Yes.
Mr Beer: Then if we look at the entry immediately below from Michael King:
“Reconciliation data provided to POL. Routing back to EDSC, does this need to be routed to Escher for a fix?”
From this record, can you see whether any action had been taken to progress any fix by Fujitsu with Escher?
Anne Chambers: Not at that point because that’s Michael King in MSU who had informed Post Office via a BIMS and now he’s routing the call back to EDSC, which is SSC.
Mr Beer: Then if we scroll down to 11:59:27 on 23 June, where Ms Obeng comments:
“Problem could have been avoided if the [postmaster] had not accepted the second TI. The Riposte Error events were apparent on SMC’s Tivoli Website, however, they took no action until the [postmaster] raised the call about the dodgy Transfers. In future, SMC would need to monitor these events and contact the office requesting that they avoid using the eventing Node and reboot it. [A KEL has now been] updated with info on what action SMC should take if event occurs during [Post Office] business hours.”
Anne Chambers: Right. Belatedly, I now have the information that there were Riposte error events which were occurring. That wasn’t mentioned earlier in the PEAK, which it very usefully could have been done, so I think this is Bug 2 in the list of bugs –
Mr Beer: Which we’re going to come to in more detail later today or tomorrow?
Anne Chambers: – which we’ll come to in more detail. But that explains why the replication didn’t happen and what the cause of this particular problem was.
Mr Beer: But this bug appears to have been picked up by reason of a call from the subpostmaster –
Anne Chambers: Yes.
Mr Beer: – not by Fujitsu’s reconciliation process?
Anne Chambers: Oh, there would have been separate calls for those as well but they haven’t been linked on here. But it does say further up in the call that the branches appeared on I think at least three of the reports.
Mr Beer: It appears to be accepted that there’s a problem with Riposte failing appropriately to pick up the transactions. We’ve seen so far the possibility of a fix by Escher has been raised. Here, Ms Obeng is suggesting that this could have all been avoided if the postmaster hadn’t accepted the second transaction?
Anne Chambers: That is strictly true but unhelpful, in that the postmaster was prompted to accept it, and did so. So I certainly would not say it was in any way his fault. Because it was not.
Mr Beer: Can we go on to page 6, please, and look at the entry for 11.17.09, “customer call”?
“Repeat call: auditor for [Post Office] site had called in to see what is happening with call.
“Advised I will call back, ringing through to SSC Barbara to find out what’s happening …
“Spoke with Barbara [and] SSC advised Catherine is on leave and she will try and get someone to look at call and call me back.”
So it seems like either the subpostmaster or an auditor in the branch is chasing now on the 6 July.
Anne Chambers: Yes, what was the previous date?
Mr Beer: Well, the call was opened on 3 June.
Anne Chambers: Right, yeah.
Mr Beer: The previous entry by Ms Obeng was 11 June and then 23 June.
Anne Chambers: Yeah. So it sounds as if nothing has been resolved as far as the postmaster is concerned, and now there is an auditor on site.
Mr Beer: Then the entry below, at 11.20.40:
“… spoke with auditor John, advised third line are going to look at the call and call back. Advised that I will call them back as soon as I can.”
Yes?
Anne Chambers: Yes.
Mr Beer: Then I think later that day, at 11.44 is the first time that you become involved?
Anne Chambers: The first time that my name has appeared on here, certainly.
Mr Beer: Yes. You say:
“I’ve checked with Mike King, the BIMS report for this problem was sent to [the Post Office] on [22 June] and should have resulted in an error notice being sent to the branch. Mike says he’ll send a note to [the Post Office] saying that the [postmaster] has been chasing this issue; I’ve asked HSH to inform the [postmaster] that they should have received an error notice and to check with the department that issues them.
“The corrected cash account that was sent still had [a receipts and payments] mismatch. The double Transfer In causes the mismatch both because of the transfer and because of the discrepancy which has been erroneously generated. The host-calculated CA” –
Anne Chambers: Cash Account.
Mr Beer: – “ignores the transfer but is still affected by the accepted discrepancy which should not have been generated. It has not really been possible to provide fully balance CA (email on this subject sent by Mik Peach to Richard Brunskill then on to John Moran, I have not seen the outcome of this).”
So you say that a BIMS notice was sent in fact way back on 22 June –
Anne Chambers: Yes.
Mr Beer: – and should have been sent to the branch?
Anne Chambers: Should have resulted in an error notice being sent to the branch.
Mr Beer: Yes. Did you speak to the auditor at all?
Anne Chambers: It doesn’t look like it, no.
Mr Beer: Can you see from this whether anyone spoke to the auditor?
Anne Chambers: Um, the helpdesk were asked to inform the postmaster but I don’t think we ever had a phone number for the auditor or I certainly didn’t have a phone number for the auditor, and HSH had been the ones speaking to him.
Mr Beer: If we can go on to 12 July, please. Thank you. If we look at the last entry for 12 July, the one timed at 12.56.23:
“An add has been sent to PinICL [and then number is given]. Tina NBSC confirms there is an outstanding error notice but she could not get exact details – she will call Paul Whittaker, the investigation officer on [and then a number is given].”
So it appears that by 12 July an investigations manager is now involved.
Anne Chambers: Um, it’s – it looks like it from what’s recorded there.
Mr Beer: That appears to be as a result of the call earlier in the day, if you just look at the top of the page we’re looking at, at 11.58.56:
“Investigation manager Paul Whittaker wants to confirm that an error notice is being sent out for the discrepancy at the [Post Office]”, and then the call was transferred to the NBSC.
Anne Chambers: Yes, because error notices were not visible in any way to Fujitsu.
Mr Beer: Then I think we’ve no need to look at it, on 5 August it’s confirmed that an error notice has indeed been issued?
Anne Chambers: Yes.
Mr Beer: Taking a step back, would this be right, and realising that you had transient involvement in the middle of this process, that a period of time between 3 June and 5 August, some three months, appears to show the subpostmaster having an auditor and an investigator in his branch at the same time that there was an identified system problem which caused the discrepancy, not him.
Anne Chambers: Yes, it does look like that, except it’s only two months, but I agree with what you’re saying.
Mr Beer: Yes, you’re quite right. Beginning of June to beginning of August, you’re quite right.
Anne Chambers: Yes, and I think, as far as we were concerned, we had informed Post Office and the ball was then in their court to produce an error notice to sort out the impact on the branch, but that obviously wasn’t – presumably wasn’t known clearly what had happened by all the people who then were involved in a branch investigation.
Mr Beer: Was that a feature of your work: that there were balls being passed between sides of the court, between – with one side of the court being Fujitsu and the other side of the court being the Post Office, and you would bat it over to the other side of the court?
Anne Chambers: I wouldn’t put it in those terms but I think we felt, when we had done our side of the investigation and notified Post Office of this, and I – then it was up to them, really, to produce the error notice and resolve the implications.
Mr Beer: Would you agree that a fair reading of these exchanges is that the only reason this was bottomed out was due, firstly, to the subpostmaster continuing to complain and, secondly, in fact, due to your intervention?
Anne Chambers: The information had been sent to Post Office saying that “This branch has made a loss as a result of this problem, which now – which they shouldn’t be liable for”. That was done well before my involvement. But whether Post Office – I don’t recall that we got any – well, it doesn’t look as if we got anything back from Post Office themselves saying “We don’t understand this, what should we do about it”, or whatever. So I can’t explain the delay in it being resolved after 22 June.
Mr Beer: Was there a system in place to ensure that if a discrepancy was identified as being caused by a systems error, that everyone on both sides of the court was aware of it as soon as possible, and that no action was taken against the subpostmaster, so that he wouldn’t, as in this example, end up with having an auditor and an investigator in his branch?
Anne Chambers: No, I think once we had felt we had done our side of things, then we wouldn’t find out what had happened between Post Office and the branch.
Mr Beer: Thank you. That can come down. Can we turn to paragraph 20 of your witness statement, please, where you address the issue of communication with subpostmasters.
Paragraph 20 is on page 5. You say you were asked to consider whether the system was adequate to manage active service tickets. You say:
“I did not consider that the system impeded our work but I am asked specifically as to whether I think there were potential changes which would have improved the systems. The problem I would identify was that the system assumed only one person or team needed to be actively working on the PEAK ticket at any one time whereas this was not always the case. If the postmaster who raised the service ticket telephoned for an update, HSD might put that request on the PowerHelp … system which they operated and that might not get copied to the PEAK, depending on which option was chosen, which might not be noticed by the developer or tester who owned the call at that moment, who would not expect to give an update to a postmaster.”
Before we explore exactly what you mean there, can you remind us of what the PowerHelp system was?
Anne Chambers: PowerHelp was the first and second line desk’s call logging system.
Mr Beer: So it was operated and viewable by only lines 1 and 2; is that right?
Anne Chambers: SSC could see the PowerHelp calls, as well.
Mr Beer: Did you habitually look at what was on PowerHelp?
Anne Chambers: Quite often, yes.
Mr Beer: You say:
“If the postmaster who raised the service ticket telephoned for an update, HSD might put that request on the PowerHelp … system and [it might or might not] get copied to the PEAK …”
Was there not an instruction to put that request on PowerHelp and to copy it onto the PEAK?
Anne Chambers: Yes, the way it worked, I can’t now remember the precise wording but, if they added an update on to PowerHelp, one sort of update automatically got copied to PEAK and the other sort didn’t, and the helpdesk didn’t always use the right category.
Mr Beer: Was anything done about that?
Anne Chambers: I think we used to remind them quite frequently to use OTI update or something, I can’t remember what it’s called, but I think I’ve seen a PEAK where we specifically say, “Make sure you” – sorry, I’ve seen a KEL or something where we specifically say, “If you have to – if you get any more information, use this particular way of doing it”.
Mr Beer: So this problem that you describe there may lead to left hand not knowing what the right hand is doing?
Anne Chambers: Yes, it can do or information just not being passed through to the people who are probably being expected to give an answer.
Mr Beer: In paragraph 21, you refer to another problem with the PEAK system. You say that:
“… if a problem had a financial impact, then it was necessary to inform the Post Office (often via the MSU team), whilst separately, it was necessary to look for the root cause(s), which might involve passing the problem to the Development team for consideration of a code fix, and also to check whether any other branches had been or might be affected by the same problem. That might result in two or more people trying to work on the same PEAK.”
Anne Chambers: Yes.
Mr Beer: What was the difficulty caused by two or more people working or trying to work on the same PEAK?
Anne Chambers: I suppose, basically, you were meant to be working on the PEAKs that were allocated to you and, if it was allocated to somebody else, then it wasn’t allocated to you. I think we saw, in that previous call that we’ve just looked at, Catherine passed the call over to Development for their comments, so Martin McConnell did some work on it, and it was only after he had then, I think, passed it back that she passed it then to MSU for them to notify Post Office of the problem.
Now, what I’ve – I might well have passed it to MSU first or, as I go on to say in the next paragraph, to clone it so you could send one call to MSU to pursue the rectification of the deficit, the discrepancy that shouldn’t have occurred, and then the second call can go on to Development at the same time, so they can start investigating the root cause.
Mr Beer: Further on this issue of passing things back to the Post Office and then down to the subpostmaster, can we look please at paragraph 42(iv), which is on page 12 of your witness statement.
If you scroll down, thank you.
This is the part of your statement where you’re describing the teams that were working in this area and you say:
“… MSU monitored Reconciliation Reports generated by the system each day”, et cetera.
Then the next sentence:
“MSU raised PEAKs so SSC could investigate cause and impact of every entry on the Reconciliation Reports.”
Then you say:
“My understanding is that MSU informed a team in Post Office of any such errors which potentially had a financial impact on a branch via a BIM report and that it was up to the Post Office to notify the branch and make any necessary correction.”
That’s essentially what you said a moment ago.
Anne Chambers: Yes.
Mr Beer: How was it possible to determine whether a problem had a financial impact or not?
Anne Chambers: Um, by us investigating what had happened, and, you know, whether it did actually affect the branch figures in any way.
Mr Beer: So you could tell for certain, one way or another, whether a problem had a financial impact on a subpostmaster or not?
Anne Chambers: Yes, I think we could. Because, you know, if the system had ended up producing figures that it would not have produced if the problem hadn’t occurred, then it’s – there could well be a financial impact.
Mr Beer: Why was it necessary to inform the Post Office of those problems, where a financial impact had occurred?
Anne Chambers: So that the branch would not be held liable for that.
Mr Beer: Was it the case that if a problem didn’t have or was judged to not have a financial impact, that the Post Office were not informed of it?
Anne Chambers: They wouldn’t be informed via that route, no.
Mr Beer: What different route would be used? What different route was used?
Anne Chambers: Something that – a different sort of a problem that had perhaps affected a lot of the estate would probably be notified to Post Office through the problem managers.
Mr Beer: Did the Post Office inform you about the action that they had taken when they were notified of a problem that had a financial impact on a subpostmaster?
Anne Chambers: No.
Mr Beer: Did you know whether they therefore told them it’s a system error, don’t worry”, or in fact they held them responsible for it?
Anne Chambers: No.
Mr Beer: Why was there no record back to Fujitsu saying what had happened?
Anne Chambers: Because that was not in the process, as far as I’m aware.
Mr Beer: Do you know why it wasn’t in the process?
Anne Chambers: No.
Mr Beer: At this time – certainly between October 2000 and mid-2010 – were you aware that the Post Office was relying on the Horizon System to produce data to prosecute subpostmasters?
Anne Chambers: I was aware that there were a small number of prosecutions.
Mr Beer: More broadly, if we look, please, at paragraph 53 of your witness statement, you address a range of situations in which the Post Office would have been informed in relation to the operation of Horizon. So that’s page 17, please. You say:
“I am asked whether Fujitsu took proactive steps to identify bugs and/or discrepancies in branch accounts caused by the same. The automatic cross-checks made and reported on the TPS, APS and banking reconciliation reports highlighted inconsistencies which might indicate a bug. These were always investigated, and MSU informed Post Office via a BIM report if the bug had affected the branch accounts, or accounts with other third parties.”
In paragraph 66, which is on page 21, this is in the middle of dealing with Bug 1, the receipts and payments mismatch bug. At the second line in paragraph 66, you say:
“Post Office would have been informed of each instance, I am not sure whether this was via a BIM report or some other route. Fujitsu would not have contacted branches directly unless the branch had raised a call in the first place.”
In paragraph 149, please. When dealing with the counter replacement bug, that’s page 44, at paragraph 149, five lines from the bottom, you say:
“Post Office would have been sent a BIMS report if there was a lasting financial impact for a particular branch but I do not know whether it was ever flagged as an ongoing problem.”
Were you personally involved in creating the BIMS reports?
Anne Chambers: No, but, as I think I say somewhere, I did at some point find out that MSU were tending to copy and paste whatever we put in the PEAK and put that onto the BIMS report. So, indirectly, yes, the words I was using were probably being passed on.
Mr Beer: So you knew that what you were saying might end up in a BIMS report?
Anne Chambers: At some point I became aware of that and I can’t remember when.
Mr Beer: So you didn’t create the report?
Anne Chambers: No.
Mr Beer: But you provided information which you knew –
Anne Chambers: Yes.
Mr Beer: – might be used –
Anne Chambers: Yes.
Mr Beer: – in the creation of a report?
Anne Chambers: Yes.
Mr Beer: What do you know about the process by which BIMS reports were created and then communicated to the Post Office? How did that happen?
Anne Chambers: That was the job of the MSU staff and I don’t know any more than that.
Mr Beer: Was there a process in place to ensure that certain types of information were always reported to the Post Office?
Anne Chambers: I don’t know.
Mr Beer: Where was the MSU sat within Fujitsu? Where was it based?
Anne Chambers: I think they were on the fifth floor at one point. I think by the time I left, they’d been sort of subsumed into SSC but – or at least sat on the same floor as us.
Mr Beer: But take the period before the advent of Horizon Online, so up to 2010. Were they essentially a separate thing?
Anne Chambers: They were a separate unit and, at some points, it was only sort of one or two members of staff, as well.
Mr Beer: What was their training, do you know?
Anne Chambers: Their training?
Mr Beer: Training and expertise?
Anne Chambers: I don’t know.
Mr Beer: Were they IT professionals?
Anne Chambers: I – not particularly, I don’t think.
Mr Beer: Were they more in the nature of administrators?
Anne Chambers: I would have said so, yes, but I had no formal involvement with them. I did spend some time with them, trying to help them understand, in particular, the way that the system was working to produce the reports, and also the flow of banking messages between the Horizon System and the banks and various other parties involved, which were all part of the massive amount of reconciliation that was done for every single banking transaction, and that was – I did do a fair bit of trying to explain that to them because then it just helped them understand what some of the different numbers on the reports meant for those entries.
Mr Beer: But the SSC was staffed with IT professionals?
Anne Chambers: Yes.
Mr Beer: You were conducting detailed investigations –
Anne Chambers: Yes.
Mr Beer: – on occasions into bugs, errors and defects?
Anne Chambers: Yes, anything over the entire very complicated Horizon infrastructure.
Mr Beer: The principal method of transfer of that information, as a result of the investigation, was in a BIMS?
Anne Chambers: Yes, going back to the specific accounting type problems, yes, that – the information about any impact on a branch was through a BIMS report.
Mr Beer: What system was there for ensuring that the right information was passed back to the Post Office?
Anne Chambers: I don’t know. This was all part of the responsibility of MSU and presumably the management of MSU and the teams that were working together and I was never a part of that.
Mr Beer: Going back to paragraph 22 of your witness statement on page 6, please. In the third line, you say:
“Looking back I can see that basing the whole process around a single PEAK (arising from an incident) per problem was not good and it is likely that there is a methodology in existence somewhere for handling this type of situation better.”
Firstly, what did you mean that the system was that the whole process was handled around a single PEAK?
Anne Chambers: Well, if you’ve – a PEAK comes in, yes, it’s a new system problem, we identify where it is, we send it off to Development do something with it. So, strictly speaking, at that point it’s off my desk. I don’t need to do any more about it. But that is not always sensible, because it’s not picking up on further occurrences, it’s not making sure we’ve got the fullest picture of it, and I just felt it would have been better if it was at that point logged as a problem, and then, if you get more occurrences, you would, you know, link the PEAKs to it so you do have that view of what it’s affecting.
And then, you know, you might have to have several actions coming off that problem for who is meant to be doing stuff to fix it, pick up the pieces, inform Post Office, and all the rest. But that was my personal view.
Mr Beer: Was that your view held at the time?
Anne Chambers: It’s certainly a view I came to while I was still working, yes.
Mr Beer: Did you tell anyone about it?
Anne Chambers: Yes.
Mr Beer: Who did you tell?
Anne Chambers: I’m pretty certain I had a conversation with Mik Peach about it. But I can’t be certain.
Mr Beer: What was the response?
Anne Chambers: Nothing, no positive outcome.
Mr Beer: What does that mean, no positive outcome?
Anne Chambers: Um, I sort of got the impression that, “Oh, well, you know, this is the way we work, this is how we’re doing it”.
Mr Beer: One might get the impression that there was a series of hermetically sealed cells within Fujitsu, the principal aim of which was to get a service ticket off their desk, and then those hermetically sealed cells were themselves rather sealed from the Post Office client. Would that be fair?
Anne Chambers: I would say to some extent that’s fair. I would also say that I personally tried to break down some of those seals a bit, and I think it wasn’t just me doing that, and I don’t think it’s, you know, it wasn’t just a matter of passing something on because we didn’t want to be holding it any more, it was passing it on because that’s where it’s got to go next because, you know, you don’t want to be sitting on a PEAK while, you know, if the next – if it’s important, obviously, that it’s investigated by Development or for it to go to MSU for – to sort out the financial consequences. So you – to that extent, you did need to pass them on in the right direction at the appropriate moment.
Mr Beer: How might that, in your view, have improved the position of subpostmasters in relation to the problems with the system that were afflicting them?
Anne Chambers: You mean, if we’d been able to – if we had – so can we go back a step?
Mr Beer: Yes.
Anne Chambers: Sorry, what –
Mr Beer: The methodology that you described of an issue being raised, an examination being undertaken to see the extent to which the problem afflicted other parts of the system and results of that investigation drawn together in a single place?
Anne Chambers: I think it would have made sure – been easier to make sure, perhaps, that Post Office, at least, were always more aware of some of these problems that were having an effect at a number of branches. How much it would actually have helped the individual postmasters, I don’t know.
Mr Beer: In paragraph 42(ii) of your statement, which is on page 11, you described, if we scroll down, please, the Horizon Support Desk.
Many of the subpostmasters who gave evidence to us in Phase 1 of the Inquiry said that they believed that the staff in the Horizon Helpdesk were using call scripts, ie that they appeared to be reading something out, some text out, some printed text. Did you ever see such printed scripts?
Anne Chambers: I can’t remember. I think for some issues where, you know – because they also handled calls about printer problems, the screen not coming on and all that sort of thing, they may have had printed scripts for some of that. I did spend a few days once in the Helpdesk office in Stevenage but I cannot remember if I saw anything actually printed out like that.
Mr Beer: Do you know who – whether they were printed scripts or scripts on a screen –
Anne Chambers: I don’t know.
Mr Beer: – was responsible for the creation of them?
Anne Chambers: I don’t know. I have no memory of ever being involved in the creation of anything like that myself.
Mr Beer: Have you got any memory of anyone else in the SSC being involved in the creation –
Anne Chambers: No.
Mr Beer: – of call scripts?
Anne Chambers: No.
Mr Beer: In the paragraph above, when you’re dealing with the NBSC, you say in the last line:
“NBSC had no direct interface with SSC, and the two teams had no access to each other’s call logging systems.”
Anne Chambers: Yes.
Mr Beer: Why did you have no access to the NBSC call logs?
Anne Chambers: I presume because they were part of a completely separate organisation, because they were run by Post Office.
Mr Beer: Would it have been beneficial for you to have had access to the actual logs to see what subpostmasters were actually saying?
Anne Chambers: Um, occasionally, at times, it might have been but, in fact, they usually had to give the same information to the Horizon Support Desk, so – but there might have been occasions where it was useful to know what else a branch had been talking about recently. But that might – there might also have been a lot of business issues that were nothing to do with – nothing to do even directly with Horizon, I’d have thought.
Mr Beer: Before the break, can we address ARP data, please. If we turn, please, to page 16 of your witness statement and paragraph 51, you say:
“… when it came to branch problems, we could only work on the basis of what was recorded on the system and any extra information which the postmaster or clerk could provide. We could not see what had physically occurred in the branch; whether for example, there was an inputting error that was not evident from the person of Riposte messages. Sometimes that meant we could not provide an answer to explain the discrepancy because none of the forms of evidence available to us gave any indication that there were no inconsistencies in the figures recorded and calculated by Horizon. In those circumstances we would have to record that we could not find any evidence of a systems error.”
Anne Chambers: Yes.
Mr Beer: We have heard described in the Inquiry a species of audit data called ARQ data?
Anne Chambers: Yes.
Mr Beer: What do you understand ARQ data to refer to?
Anne Chambers: ARQ data was all the messages written on the branch counters, as they replicated to the data centre, were also streamed off into a large series of very, very large files, which were then, end of each day, can’t quite remember, but in some way sort of securely locked and then could only be accessed again through the special audit systems.
If an audit extract was requested by Post Office, they would have to say the branch that they wanted and the date range, and the standard information, I think, that POL got, Post Office got for one of these enquiries was – well, it was two spreadsheets, it ended up as, and one was the transaction data messages and the other was the Riposte event messages.
So they would be, I think, all in date order or it may still have been in the date as received at the data centre. So it was a subset of the whole lot but it was all the transaction data. It was also possible to get an unfiltered set, which would be all these messages in their entirety, and I –
Mr Beer: Did you say all messages?
Anne Chambers: All messages, yeah. I’m talking here about Legacy Horizon, it was very slightly different for HNG-X.
Mr Beer: So transaction data messages, Riposte messages and also possible to get an unfiltered set of all messages?
Anne Chambers: Yes, transaction data, Riposte events, that’s log on/log off, those sort of things. But it was possible to have the unfiltered set.
Mr Beer: It’s been suggested by some witnesses that the ARQ data equated to data that recorded every keystroke made by a subpostmaster in branch?
Anne Chambers: That was never captured.
Mr Beer: Simplifying matters, was it therefore the data when a subpostmaster committed what they had done to the stack, essentially?
Anne Chambers: Yes, because the messages got written to the message store, the transaction messages were written to the message store when the basket was settled.
Mr Beer: Where was that data stored?
Anne Chambers: At what point? Do you mean the ARQ data?
Mr Beer: Yes, the ARQ data?
Anne Chambers: On secure servers somewhere.
Mr Beer: Do you know where?
Anne Chambers: No.
Mr Beer: For how long was it stored?
Anne Chambers: I don’t know, because that was never my responsibility. But it was lots of years.
Mr Beer: You say in this paragraph here, you couldn’t see what had happened in branch, whether there was an error inputting that wasn’t evident from a pattern of Riposte messages –
Anne Chambers: Yeah.
Mr Beer: – and therefore you were left with no evidence of a systems error.
Anne Chambers: Yeah.
Mr Beer: In those circumstances, couldn’t you use the ARQ data?
Anne Chambers: No, because the ARQ data – well, I’m not saying here that we couldn’t see what had happened in the branch. We could see from the message stores that we looked at the transactions messages being committed, and so on, and we could also see the extra messages that gave us some extra clues as to what had been going on sometimes. But the ARQ data was just that, that same data that’s stored in a different place.
Mr Beer: It wouldn’t have shown you anything more at all?
Anne Chambers: No.
Mr Beer: So is that why you didn’t obtain such data, ARQ data, when investigating what I’ll call the simple or common or garden PEAK?
Anne Chambers: Yeah, there was no – we felt there was no need to go to the ARQ data because we had the same information that we could get out of the message store on the correspondence servers.
Mr Beer: So your view was that there was nothing additional in the ARQ data –
Anne Chambers: No.
Mr Beer: – than the data that you had access to would have shown you?
Anne Chambers: It was exactly the same data.
Mr Beer: Was there a cost to obtaining the ARQ data?
Anne Chambers: Um, I believe it was something that Post Office were charged for or they had a certain number that they could ask for in a year. But, above that, they had to pay.
Mr Beer: Did that have any effect on the extent to which ARQ data was obtained?
Anne Chambers: Not as far as I was concerned. If I was investigating something that had happened a very long time ago and I needed to look at that old data, then I would ask for it and would get it. And I don’t recall that anybody ever tried to charge me for it or suggested that I should be charged for it.
Mr Beer: In what circumstances would you therefore obtain ARQ data?
Anne Chambers: If, for some reason, I was specifically asked to go and look at something that had happened some time – a long time ago. I can’t now, except for one instance, remember doing it as – it certainly wasn’t something that was done as a matter of course, because normally, if something had happened a very long time ago, we wouldn’t – probably wouldn’t be expecting to investigate it.
Mr Beer: The one incident you can remember, is that because there may have been others that you’ve forgotten or it’s likely that, in your 16 years, you only had cause to access ARQ data once?
Anne Chambers: I have a feeling it was more than once, but I can’t – it was possibly in relation to prosecutions, and so on, or some investigation that somebody wanted doing, but I cannot now remember much by way of detail.
Mr Beer: Why would you access that data, rather than the data that you would habitually use in the case of a prosecution?
Anne Chambers: Because it had – well, that would be – for anything that had happened more than a month, five weeks ago, you would have to go – you wouldn’t have any other sources of data, except the ARQ data.
Mr Beer: It was the only place you could find it?
Anne Chambers: Yes.
Mr Beer: Thank you. On that note, sir, might that be a convenient moment?
Sir Wyn Williams: Yes.
Mr Beer: It’s 3.10 now, I think.
Sir Wyn Williams: So 10 minutes or 15? Which do you think?
Mr Beer: Knowing where I am in my notes, ten minutes, please.
Sir Wyn Williams: Fine.
Mr Beer: Thank you.
Sir Wyn Williams: Ten minutes, that’s 3.20 then, is it, Mr Beer?
Mr Beer: Yes, thank you.
(3.10 pm)
(A short break)
(3.21 pm)
Mr Beer: Thank you, sir.
Can I turn to the tenth topic under this heading then, which is attributing a problem to user error and doing so as a default option. The Inquiry, Mrs Chambers, has heard some evidence on the default use of user error within Fujitsu in the absence of any other explanation.
I want to look at some materials, please, that address the question of attributing a fault to user error or a discrepancy to user error. Can we start, please, with FUJ00082302. Thank you. We can see this is “KEL acha”, which I think is one of yours, Anne Chambers –
Anne Chambers: Yes.
Mr Beer: – 1717T. It’s raised by you, we can see, in the fourth line, on 30 July 2010, and closed or last updated by you on 10 February 2015. We can see from the “Symptoms” section of the KEL that:
“The branch complained that they had a loss of £240 on one day. NBSC had been unable to find any reason for the discrepancy so the call was sent to Horizon to check for system errors.”
Then scrolling down, “Problem”. You say:
“A discrepancy is the difference between the cash the system thinks the branch should have, based on previous balanced figure and the transactions recorded since, and the cash declared by the branch.
“Possible causes:
“[Postmaster] has made an incorrect declaration.
“Transactions as record on the system do not match what actually happened at the branch.
“Outstanding recover.
“Withdrawn products.
“Known system problem (these should be monitored for, or be easy to spot from events, etc)
“Unknown system problem.
“Solution …”
Scroll down, thank you:
“Make sure NBSC have already investigated and include details of what they investigated and found on the call.
“Which stock unit is affected?
“Trading period/balance period?
“Which day did the discrepancy occur?
“How much is it?
“If the discrepancy was apparent only when they balance, when, before this, were figures correct?
“Do they do a nightly Variance Check after declaring cash?”
Then you put a “NOTE” in, in capitals:
“SSC do not accept credible calls until all information on the investigation performed by NBSC is detailed and they have advised why they believe there is a software fault.
“Solution”, at the foot of the page:
“When responding, if they have given specific examples that you can explain, do so; otherwise make clear it is not a system problem (assuming you have checked that everything adds up). See [PEAK ending 446] for an example response which may help with the wording.”
So is a summary of what you’re saying in this KEL to the earlier lines of support, “Do your homework, follow the proper processes before referring a discrepancy up to us in the SSC”?
Anne Chambers: Yes.
Mr Beer: It was therefore an attempt, was it, at discouraging them from, in some cases, sending cases up to the SSC?
Anne Chambers: I don’t think we were discouraging them but we wanted them to get as full a picture as they could of whatever information was available about the circumstances, which we could then use to inform our investigation.
Mr Beer: If we go back to the bottom of page 1, under the note, you’re effectively telling them “You’ve got to believe that there is a software fault before you refer it on to us”?
Anne Chambers: We certainly felt that NBSC should help the branch to rule out any likely user errors.
Mr Beer: Well, is the premise of my question right, that you were telling the earlier lines of support that, before referring a problem on to SSC, they should believe that there is a software fault?
Anne Chambers: Well, that was what we were there to investigate, was software faults.
Mr Beer: So it’s a yes?
Anne Chambers: So yes. But obviously they could not tell definitively if there was one or not.
Mr Beer: That’s what I was going to ask you. How could the HSH or the SMC identify or analyse whether there was a software fault?
Anne Chambers: They weren’t asked to do that. They were only asked if they believed there was a software fault and a belief is not the same as proof.
Mr Beer: So what quantum of proof did they need, what quantum of belief did they need?
Anne Chambers: Well, they didn’t. They only needed to believe it. We wanted them to have done – I think it was felt that we shouldn’t be investigating these types of calls, given that a discrepancy is most likely to be caused by an issue at a branch. I’m certainly not saying that’s the only cause but, overall, I think most branches would accept that, from time to time, they would have a discrepancy of some amount, and so we did want to make sure that the normal checks had been made before it came over to us, really as a last resort.
Mr Beer: Amongst the possible causes that are listed is “unknown system problem”?
Anne Chambers: Yeah.
Mr Beer: I think it follows that the first two lines of support would not be able to identify if there was an unknown system problem, would they?
Anne Chambers: No.
Mr Beer: So were you expecting them to investigate whether the discrepancy that they had been told about was the consequence of a known system fault?
Anne Chambers: We more wanted them to try to rule out discrepancies caused by business errors at the branch.
Mr Beer: By business errors, you mean?
Anne Chambers: Accounting errors at a branch. Well, not accounting errors but, you know, could they have mistyped an entry somewhere?
Mr Beer: Then over the page, in the instructions to, I think, the SSC –
Anne Chambers: Yes.
Mr Beer: – at the foot, you say “When responding”. So that’s when you, SSC people are responding to the lower lines of support:
“… if they [that’s the lower lines of support] have given specific examples that you can explain, do so …”
Anne Chambers: Yes.
Mr Beer: “… otherwise make clear it is not a system problem (assuming you have checked that everything adds up).”
Anne Chambers: Yes.
Mr Beer: What does that mean –
Anne Chambers: I –
Mr Beer: – “make [it] clear it is not a system problem”?
Anne Chambers: Once you’ve checked everything and have not been able to find any sign of any system problem. So I think I explained before that, you know, you’ll get your running totals of the system cash position and you’ll compare it with what the system had calculated its position was at a particular point in time. You’ll compare the two, if those two weren’t equal then, yes, you do have a system problem and then you’ve got to find out if it’s a known one or an unknown one. But if the two figures are in step, the system has calculated the discrepancy correctly, then you’ve got to try to see which transaction or set of transactions might not be right, which is leading to the discrepancy and, if you can identify something that looks suspect, then could that have been caused by a system error?
And if you can’t find anything looking suspect, you can’t find any events, anything else indicating any sign of a system problem, then you are left with the fact that the system thinks there should be so much in the drawer, that the postmaster is saying there’s a different amount in the drawer, the only reason – the only way you’re going to find why those two amounts are different is if you can identify something recorded on the system that doesn’t match what actually took place at the branch.
Mr Beer: Didn’t that create a default position that, in the absence of evidence of a system error being provided by the subpostmaster themselves and the absence of evidence of a problem fitting with a known existing system error, the problem would be written up as user error?
Anne Chambers: Or an unknown system error. I mean, if you’ve got an unknown system error, it’s going to have to have left some trace somewhere in the numbers that will give you a clue as to where it is, what sort of transaction. But unless – but there’s no way that somebody investigating that problem with no knowledge of what has been carried out at the branch can say “That there doesn’t match what they actually did at the branch on the day”. So to that extent, yes, we were dependent on the postmaster being able to say, “Hang on, why has that there gone in for £2,500 when it was only £250?”
Me investigating, I’m not going to be able to know that it should have been £250 but actually it’s £2,500.
Mr Beer: What weight was accorded to the statement by a subpostmaster who had proactively raised with the Post Office and with Fujitsu a discrepancy that he hadn’t, in that example, put in the £250?
Anne Chambers: So you’re saying, so if he is saying “Look, it says £2,500 but I only put in £250”?
Mr Beer: Yes.
Anne Chambers: If he can say it’s that line there then we would look at everything to do with that line there to see if we could see any signs of it being a system problem that had caused it.
Mr Beer: But what if it came to him saying “No, no, no, it says £2,500 there, only £250 – it was only £250?” Where he’s the one that’s proactively raised this by calling in, what weight was accorded to what he was saying?
Anne Chambers: Quite a lot of weight. We wouldn’t – we wouldn’t automatically say, “Oh, you must have got it wrong”. But if he’s then got to that point where he can identify it, then we can at least do something about the financial side of it, in that he now knows which line is wrong, and we can attempt to pass this information through to Post Office so that that line, whatever it was, could be rectified in his accounts, which, to some extent, is the important thing to be done in the short-term.
Longer term, if you’ve had a lot of branches all reporting that precise type of problem, you would have to try to work out how come it could be wrong but, in practice, I don’t recall anything specifically like that.
Mr Beer: Can we take a look at a practical example. You’ll see that this KEL mentions at the end there the PEAK ending in 446.
Anne Chambers: Yes.
Mr Beer: Can we look at that PEAK, please? It’s FUJ00083493. You should see that’s a PEAK ending in 446. It’s dated or raised on 12 November 2013 and the problem is described in the PEAK “Summary” as:
“[Postmaster] doing cash declarations every now and again has major loss.”
Yes?
Anne Chambers: Yes.
Mr Beer: We can see from the “Progress Narrative” at the first entry, exactly the same is entered. Then if we scroll down to the foot of the page, we get a bit more detail. Do you see the entry at 13.40.00? It’s about halfway down this page. Yes, that one. Thank you:
“[Postmaster] has had cash declaration problem throughout year and is losing a lot every now and again.
“He phone up Helpline told him can’t of [sic] declared properly.”
I think that means he phoned the helpline who told him that he can’t have declared properly.
Anne Chambers: Yes.
Mr Beer: “He states that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing money.
“[Postmaster] stated he has 3 post offices, only happens at this site.
“Advised [I would] get this call sent to PEAK to look for any software error.”
Then a little further on:
“… only done cash declaration on back office counter, hasn’t tried on any other counter.
“done a declaration this morning and had a £6,000 loss.
“It shows no error message when doing it.
No report prints out, only printout of cash declaration.”
Then if we go over to page 2, please, and look at the entry about five or six lines in at 14.01.00, thank you:
“Voiced [I think that made a voice call to] NBSC [quoting a reference] to see what checks they have done themselves before transferring call to Horizon.
“They stated they had trainers come into the office and ruled out user error.”
I just ask you to remember that line, please.
Then on the same page, at 14.57.05, so that’s – yes, in the middle of the page now:
“NBSC called in for a message for Rob.
“NBSC states that user error checks were carried out by auditors at the site and not over the phone.
“Advised I would update the call.”
I’d ask you to remember that message too.
Then we come to your entry, same day, at 15.50.04, and you say:
“When the variance is checked following a cash declaration, the system compares the declared amount against the expected amount (ie the opening cash figure for the period adjusted by all cash movements since then).
“I have checked the system figures … in the declared figures for stock unit AA since the rollover on 7 November and can confirm that all the variants reported since then have been calculated correctly. There are no known issues that would result in the variance being incorrect.
“For example, on 7 November, the opening cash figure rollover was £60,125.
“£22,300 was remmed in.
“£6,500 was transferred out.
“Single SC transaction for £11,183.60 in.
“So at this point there should be £87,038.60 in the drawer.
“Cash declared at 15.29: £81,580 ie £5,528.60 short (though the variance wasn’t checked at this point).
“I can’t tell why the declared cash doesn’t match the expected figure, the branch need to make sure that what they have recorded on the system is correct, and investigate the anomalies.
“Please pass this back to NBSC.”
Yes?
Anne Chambers: Yes. I did also cover this in my witness statement.
Mr Beer: Yes. Now, the final part of your conclusion there is that there were no known issues that will result in the variance being incorrect and:
“I can’t tell why the declared cash doesn’t match the expected cash … the branch need to make sure that what they have recorded on the system is correct”, and that they should investigate the anomalies.
Anne Chambers: Yes, yes.
Mr Beer: The NBSC had got the trainers to go physically to the branch to establish what had occurred, and, in particular, whether that was a case of user error, hadn’t they?
Anne Chambers: Yes.
Mr Beer: That was significant information, wasn’t it?
Anne Chambers: Yes, it was, and I took that on board.
Mr Beer: Auditors had gone to the branch for the same purpose, rather than seeking to find out what had gone wrong on the phone, hadn’t they?
Anne Chambers: It appears so, yes.
Mr Beer: Both had come back and said, “This is not a case of user error”, hadn’t they?
Anne Chambers: Yes.
Mr Beer: So how –
Anne Chambers: I –
Mr Beer: – please – was the subpostmaster going to investigate the anomaly?
Anne Chambers: I don’t know. I felt by giving some specific examples of a specific day, where actually very little had happened, but they were already £5,500 short apparently. If they could compare those three, four figures that I gave, perhaps they would be able to see which of those was wrong in that process, in what I could see.
And, although I only gave the figures from that very simple point from the beginning of November, from the beginning of the period, I had actually done this check that I’ve already described and, in every case where the system cash figure was calculated, it was in line with what it should have been. So it wasn’t that messages were not being picked up and included in the accounts; everything seemed to be there. The branch was declaring how much cash they’d got when every time they did that, they’d got a report printed out that showed the cash declaration.
Something was not right. And, yes, the numbers were jumping about all over the place. The discrepancies were varying because they kept declaring wildly different amounts of cash.
Mr Beer: Was it them declaring wildly different amounts of cash or the Horizon System?
Anne Chambers: It definitely did appear to be them and I did, of course, wonder if it could be something that was wrong with the declaration mechanism but, again, they got the report printed out that showed them what the declaration was and I would have expected that if they had declared – so they had to declare by denomination, I would have expected that they would quite easily be able to see if what was on their reports didn’t match the piles of cash that they had sitting in front of them.
I’m not happy – I’m not very happy with this PEAK. I don’t think I handled it terribly well. I was frustrated by it and I think that shows, and I was still frustrated by it when I wrote the bit in the witness statement, because, you know, it really looked like there was a genuine problem. There was no sign of it, the figures, you know, when you added everything up, yes, the discrepancy based on what they’d done, what they declared, the discrepancy was apparently correctly calculated. Well, it was correctly calculated.
So what was it in the data that was going into these calculations that wasn’t right? And I couldn’t see it. And, you know, when you’ve got three transactions and, at the end of it, they’re out by £5,500 but you’re adding up the numbers. It’s not a matter of my belief or not; it’s objective. That was what was there. So something – something definitely wasn’t right but couldn’t see it, without being in the branch.
Mr Beer: An auditor and a trainer had been in.
Anne Chambers: Yes, I wish I could have worked with them to try to get to the bottom of this but that was not – as far as I was aware, that was not an option.
Mr Beer: If we go back to page 1 at the top, please. You closed this call off, top right, four lines in, “Closed – No fault in product”, didn’t you?
Anne Chambers: I couldn’t find a fault in the product.
Mr Beer: Because you couldn’t find one, are you therefore saying this was user error by the subpostmaster?
Anne Chambers: Not necessarily user – well, I didn’t know what it was. Perhaps I should have just put – I can’t remember if there was an unknown category.
Mr Beer: Well, maybe look at page 3, please, top box. 15.50.04, entered by you:
“Defect cause updated to … General – User.”
You’re saying it’s a user error, aren’t you?
Anne Chambers: I’m saying I was very frustrated by it because, if you like, I wanted to find a system error.
Mr Beer: That doesn’t come across at all in this note, does it? You’ve closed this off saying there’s no fault in the product –
Anne Chambers: Yes.
Mr Beer: – the defect is the subpostmaster, haven’t you?
Anne Chambers: That’s – that’s how it comes over, yes.
Mr Beer: Well, is there a different way of reading it?
Anne Chambers: Um, there must be something that they could have spotted at the branch to give us a clue as to what was happening.
Mr Beer: In the light of what the trainers and the auditors had found in the branch, how could you possibly say there was no fault in the product and this was a user error?
Anne Chambers: Because the numbers added up.
Mr Beer: Can we look at your witness statement, please, at paragraph 183, which is on page 52, where you were asked about this entry, and eight lines in – eight or ten lines – you say:
“This could be user error or potentially system error but could only be identified by the branch.”
Can you see that?
Anne Chambers: Yes.
Mr Beer: Here you seem to accept that this could be a user error or a system error?
Anne Chambers: Potentially.
Mr Beer: Why did you write it up as no fault in the product and that it was a user error?
Anne Chambers: Because I was rather frustrated by not – by feeling that I couldn’t fully get to the bottom of it. But there was no evidence for it being a system error.
Sir Wyn Williams: Does that mean there was no evidence that you saw? What I’m trying to get at is the use of the phrase “Unknown system error”, all right? There are two ways I could take that: one is that there is evidence of a system error but you can’t identify the root cause; the other is that there is a system error, on balance of probability, but you simply don’t know what it is?
Anne Chambers: I felt that there was just no – something was obviously wrong, in that the branch obviously were getting these discrepancies that they weren’t expecting, but all I could see on my side was that they were apparently declaring these differing amounts, and I certainly knew – didn’t know of any system errors that would cause that to happen, or to cause – or that would take what they were declaring and not record it correctly. I wasn’t – and I couldn’t see any signs of that happening and nobody was saying, “Look, we declared this but on the report it says that”.
So I felt, on balance, there was just no evidence of a system error.
Sir Wyn Williams: But as I’ve understood the line of questioning that Mr Beer has been putting to you, it’s not just that the subpostmaster was making the error whatever that was, but the auditor and the trainer were making the same error; is that it or have I misunderstood it?
Anne Chambers: Well, yeah, I – yes, I don’t know why the – I’d have thought, if the report was showing something but then on the system it was recorded as a different amount, I would have expected the auditor or the trainer to have spotted that. Yeah, I mean, now, I feel maybe I could have made steps to try to talk to the auditor or the trainer and to try to talk through it a bit more. But, no, I’m not happy with this one.
But I still stand by there being no indication of a system error and the numbers that they were recording just didn’t make a lot of sense.
Mr Beer: You’ve been told by the PEAK itself that the problem was a recurring one.
Anne Chambers: Yeah.
Mr Beer: It had happened throughout the year?
Anne Chambers: I’m not sure remembered that, but –
Mr Beer: That was in the PEAK that we looked at?
Anne Chambers: Yeah.
Mr Beer: Did you think how likely was it that this was a persistent human counting error by the subpostmaster or did you think that the very report itself of a repetitious error was itself suggestive of some system fault?
Anne Chambers: I think I thought that it just looked possibly like a certain amount of carelessness. I don’t know. I couldn’t explain it. But I also, you know, perhaps in my knowledge of the whole system, couldn’t understand, you know, this – a problem like that just affecting one branch. That’s very unlikely. But that’s – that was probably at the back of my mind.
Mr Beer: What checks did you make to see whether it had affected any other branch?
Anne Chambers: I’m not aware that around this same period of time any other branch had knowingly reported a similar problem.
Mr Beer: I asked a different question: what checks did you make to see whether it had affected any other branch?
Anne Chambers: I don’t know that I made any direct checks but at that point I would have seen any calls like that that were coming in to SSC.
Mr Beer: You knew by this time, 2013, that it was not always possible immediately to find a fault in the system, wasn’t it? You knew that from your past experience of system faults with Horizon? Much investigatory work needs to be undertaken?
Anne Chambers: You can usually see evidence of a fault. You might not be able to get to the root of it very easily but you can usually – if a system fault has occurred, there’s normally some sign of it, specifically that the numbers don’t add up.
Mr Beer: We saw that this call was allocated to you at 2.34 pm and that you wrote it up as “Closed – No fault with product, user error”, at 3.50 pm.
Anne Chambers: Mm-hm.
Mr Beer: So assuming that you picked up the call at exactly the moment it was assigned to you, you spent 1 hour 14 minutes on it?
Anne Chambers: Mm-hm, yes.
Mr Beer: What did you do in that time?
Anne Chambers: I would have done more or less what it said in the PEAK – sorry, the KEL acha17170: extracted the transaction data from the branch database, which is what had replaced the message stores; added up the cash; checked the variance calculations matched the system figure at each point.
Mr Beer: Over what period?
Anne Chambers: Since the 7 November, I think it was. The best part of a week. That was the week they’d given the specific examples for.
Mr Beer: What about what the subpostmaster had said in the PEAK, that this has been a problem throughout the year?
Anne Chambers: I probably didn’t go back before –
Mr Beer: I’m sorry, you probably –
Anne Chambers: I probably didn’t check back beyond the one week for which they had given sort of specific examples about it varying by £2,000 and £4,000.
Mr Beer: Why not?
Anne Chambers: Because I wasn’t sure it would give me any more than showing that, you know, these – their declarations seemed to be varying so wildly.
Mr Beer: Does what you’ve done, as displayed by this PEAK, indicate your more general approach, to the effect that if you couldn’t positively see a system error, the default conclusion that any discrepancy is the fault of the user?
Anne Chambers: If there’s no indication at all of a system error then, yes, that’s the inference that you draw from that.
Mr Beer: Was that the operating approach of others within the SSC: unless there’s positive evidence of a system error, it is the subpostmaster’s fault?
Anne Chambers: We weren’t necessarily assigning blame. We were saying there was no indication of a system error.
Mr Beer: Writing off a PEAK, “user error”.
Anne Chambers: In this case, yes, which I’ve already said I probably shouldn’t have done.
Mr Beer: In the Horizon Issues trial, the court was given evidence that between 1 January 2011 and 31 December 2014 the SSC received 27,005 calls, meaning that, on average, there were 563 calls per month.
Anne Chambers: Mm-hm.
Mr Beer: Does that sound about right: between 500 and 600 calls or tickets a month?
Anne Chambers: It sounds like quite a lot but it’s probably about right.
Mr Beer: The Inquiry has heard evidence that the SSC was under pressure to close PEAKs because of the pressure of work. Is that how you felt when you worked in the SSC?
Anne Chambers: No. I mean, how many of those calls were counter related?
Mr Beer: The evidence doesn’t disclose that. Was there any pressure at all on you to close PEAKs prematurely?
Anne Chambers: No.
Mr Beer: In paragraphs 27 to 29 of your witness statement, there’s no need to turn them up at the moment, you address the prioritisation of tickets within the SSC.
Anne Chambers: Mm-hm.
Mr Beer: Were there any financial implications in respect of the prioritisation of tickets?
Anne Chambers: I was trying to remember and I think in there I say, yes, there may have been something but couldn’t remember it. I think I’ve seen – read more documents since which have reminded me that some types of call – some of the reconciliation calls involving sorting out where people’s bank accounts might be adrift or where bill payments might not be going through, those had to be resolved pretty quickly.
That meant making sure that the financial implications were resolved, but if there was a root cause that needed further investigation, that would still be done after the financial side of things was dealt with. So, yes, we were aware some calls you couldn’t leave lying around for long but that didn’t mean that they had to be closed.
Mr Beer: We can take the witness statement down from the screen. Thank you.
How quickly did the different priority tickets need to be resolved before any financial penalty was imposed on Fujitsu for late resolution?
Anne Chambers: Um, I can’t remember the details. As I said, I don’t think it was all calls; I thought it was just a certain type of calls. And a priority was probably within 24 hours to get the financial side of it sorted.
Mr Beer: If a ticket wasn’t given the priority that you thought it needed, would you formally change the priority or would you just get on with the ticket?
Anne Chambers: It would depend where it was going. If it was something that I was dealing with, I might not change it, but I would still treat it as I thought it needed doing.
Mr Beer: Were performance statistics kept of your team, in terms of resolution?
Anne Chambers: You’d have to ask my manager.
Mr Beer: Were you aware, in the 16 years you worked there, that performance statistics were kept?
Anne Chambers: No.
Mr Beer: Was there any element of performance-related pay in the SSC?
Anne Chambers: No.
Mr Beer: Can we turn, please – the last topic for today, I think – to remote access. I want, please, to look at the nature and extent of remote access within the SSC, the extent to which individual branches were notified of issues that affected them and who was responsible for such communications.
Can we start, please, with FUJ00088036. You’ll see this is a document entitled “Secure Support System Outline Design”.
Anne Chambers: Yes.
Mr Beer: It’s dated 2 August 2002, so about two years into your tenure at the SSC?
Anne Chambers: Yes.
Mr Beer: At the foot of the page, you can see that one of the approval authorities is Mik Peach?
Anne Chambers: Yes.
Mr Beer: But you’re not on either the distribution, the approval or the reviewer list, okay. Was this the kind of document that would be filtered down to you?
Anne Chambers: Um –
Mr Beer: Just go back to the top.
Anne Chambers: Yeah, I have no recollection that I ever saw it.
Mr Beer: That could be because 21 years have passed.
Anne Chambers: It could well be, yes.
Mr Beer: But is it the type of document that would be provided to you?
Anne Chambers: It would probably be input on our website for us to read –
Mr Beer: So an intranet?
Anne Chambers: – which isn’t the same thing – so yes, an intranet. But, as I say, I’ve no idea whether I did read it at the time.
Mr Beer: Can we go to page 15 of the document, please. Let’s see whether what it says is correct. It’s paragraph 4.3.2, “Third line and operational support”:
“All support access to the Horizon systems is from physically secure areas.”
You told us, I think so far as the SSC is concerned, that’s correct?
Anne Chambers: Yes.
Mr Beer: “Individuals involved in the support process undergo more frequent security vetting checks.”
Were you aware that you underwent more frequent security vetting checks than some other unnamed people?
Anne Chambers: No. I know I was vetted when I first joined, but I wasn’t aware of –
Mr Beer: Being re-vetted –
Anne Chambers: – being re-vetted.
Mr Beer: – in the 16 years afterwards?
Anne Chambers: Yes.
Mr Beer: “Other than the above, controls are vested in manual procedures requiring managerial sign off controlling access to post office counters where update of data is required. Otherwise third line support has:
“Unrestricted and unaudited privileged access (system admin) to all systems including post office counter PCs;
“The ability to distribute diagnostic information outside of the secure environment; this information can include personal data (as defined by the Data Protection Act), business sensitive data and cryptographic key information.”
Is it correct that at this time, 2002, you in the SSC third line support had unrestricted access to all systems including post office counter PCs?
Anne Chambers: If that’s what it says, yes.
Mr Beer: Was it your experience that that is correct?
Anne Chambers: We had access. I didn’t ever test it to find how unrestricted it was.
Mr Beer: It says that such access was unaudited. Was that correct as well?
Anne Chambers: To the best of my knowledge, yes.
Mr Beer: The paragraph continues:
“The current support practices were developed on a needs must basis; third line support diagnosticians had no alternative other than to adopt the approach taken given the need to support the deployed Horizon solution.”
Was that your understanding of why you had been given unrestricted and unaudited privileged access to all systems? That it wasn’t planned, it wasn’t part of the design, it wasn’t the architecture; it was reactive to events?
Anne Chambers: I don’t – I’m not sure that I thought about it. I joined the SSC and I was shown, you know, as I said, you know, “If you need to access a counter, this is how you connect up to it”, and so on. These were systems that I wasn’t particularly familiar with, so I just did what I was told on the occasions when I did need to access a counter.
Mr Beer: So you didn’t address your mind particularly to how it had come about that you had –
Anne Chambers: No.
Mr Beer: – such unrestricted and unaudited access to –
Anne Chambers: No, I –
Mr Beer: You just took it as given that you got it?
Anne Chambers: Yes. I probably didn’t question whether it was audited or unaudited at that point. It was something that was needed at times to do our job.
Mr Beer: Can we move forwards a year then, please, to FUJ00088082. This is a document described as a “Support Guide” for the OpenSSH. Can you recall what the OpenSSH was?
Anne Chambers: I do remember at some point the way we accessed counters, and other systems as well, changed, and SSH rings a bit of a bell. So yes, this is what we switched to.
Mr Beer: So this document again is not signed off by you, reviewed by you or, on its face, distributed to you, but the process would have been something that you would have been familiar with by June 2003; is that right?
Anne Chambers: I imagine so, because we would have changed to using the method that’s documented in here.
Mr Beer: We can see the date on the top right of 30 June 2003. If we go to page 6, please. It appears that this was an attempt to – I’m going to call “sort out” the unprivileged – sorry, the unaudited and unrestricted privileged access that the SSC had. Can you see in the second paragraph of the introduction it says:
“It is necessary, for security and auditing purposes, to provide a method that allows datacentre and counter systems to be managed interactively but for all these management actions to be captured. When these actions have been captured (or logged) it must be possible to audit the actions. This, in turn, means the logs must be in an easily understandable format.”
Anne Chambers: Yes.
Mr Beer: The report then goes on – I’m not going to go through it – to explain the different ways the system produces an audit trail for all actions performed during what I’m going to call remote access.
Anne Chambers: Yes.
Mr Beer: What was your understanding of the process that had to be undertaken after the introduction of this process in June 2003?
Anne Chambers: Um, I think I was aware that everything would be captured and logged somewhere, but I don’t know – didn’t know and don’t know any more details than that.
Mr Beer: Can we go, please, to FUJ00138355. Can you see that this is a document created by your manager, Steve Parker, on 8 September 2011 –
Anne Chambers: Yes.
Mr Beer: – and was last updated by somebody called Adam Woodley in February 2021?
Anne Chambers: Yes.
Mr Beer: You’ll see what Mr Parker said:
“GDPR regulations require that access to personal data remains within the European Union and PCI data security standards mandate physical security restrictions must be applied where [data] access is allowed to user data. Currently the only units which fulfil all these requirements for data access are the SSC and ISD Unix.”
Do you know what ISD Unix was?
Anne Chambers: I think they were the operations team based in Belfast.
Mr Beer: “The responsibility for data correction is vested in the SSC although ISD sometimes act under SSC authorisation (via the application of a tested script).
“Corrections to live system data must be authorised via Account change control and auditable. Any correction requiring APPSUP role is to be witnessed by a second member of the SSC. Both names must be recorded on the change control for audit purposes.”
Anne Chambers: Yes.
Mr Beer: This appears to describe in 2011 the so-called “four eyes” approach, or “four eyes” method, namely one person from the SSC logs on to the counter and makes the – I’m going to call them corrections to the branch’s transactions whilst another person watches, doesn’t it?
Anne Chambers: Yes.
Mr Beer: They were supposed to write a narrative account of what they’d done and record the names of both people, weren’t they?
Anne Chambers: Um, yes. I can’t remember if the second person was meant to add their own name themselves, but I can’t remember.
Mr Beer: Just look under “Data corrections”:
“Support investigation may indicate the need for data correction. In this context data correction is any support action that results in the modification or removal of Post Office data. If any correction is required then details must exist in the form of a narrative on, or attachments to, a PEAK incident to provide a clear audit trail. An approved Account change control entry … must also exist and be cross-referenced from the PEAK.”
Then “Financial Data”:
“Changes to financial data are rarely required. Where a requirement exists, such changes must be made via contra journal entries to maintain audit trail and the change must be made using the two man rule. The ‘two man rule’ (sometimes called the ‘four eyes rule’ in security circles) specifies that there must be two individuals that must act in concert in order to perform the same action. Further, each individual should have comparable knowledge and skill in order to detect attempts of subversion initiated by the other.
“Within the SSC, one member of the SSC will perform the data correction whilst a second member of the SSC will witness the change being made. Both names must be recorded on the change control for audit purposes.”
Anne Chambers: Yes.
Mr Beer: So this process depends on people in SSC depending – or carrying out faithfully and recording what they’re supposed to have done?
Anne Chambers: Yes, or what they did do.
Mr Beer: Yes.
Anne Chambers: Yes.
Mr Beer: You remember the 2003 document that we looked at. I didn’t take you through its details, but do you recall that that, if it had been carried into effect, would have required that a securely managed ID user had been recorded and logged in real time against each and every action performed, rather than an after-the-event narrative description of what we had done?
Anne Chambers: I’m sorry, I don’t remember that document. That was the SSH, OpenSSH document?
Mr Beer: No, it was the document that I took you at some speed through, in the interests of time.
Anne Chambers: Sorry.
Mr Beer: We can go back to it. It’s FUJ00088082.
Anne Chambers: Yes, the OpenSSH –
Mr Beer: You remember that?
Anne Chambers: Yes, it was the OpenSSH Support Guide.
Mr Beer: If you turn to page 8, I’m afraid you’re just going to have to read that quietly to yourself, the “Basic Procedure”.
Anne Chambers: That procedure is just talking about creating the users, not the users using the systems.
Mr Beer: Yes. Then if you go on to page 10 on the “Logging Server” and scroll down, and then look at 5.2.
Anne Chambers: Yes. Sorry, I can’t …
Mr Beer: Sorry, you said you can’t?
Anne Chambers: I’m not sure what I’m meant to be looking for in here.
Mr Beer: Okay, then go on to 5.3.
Anne Chambers: Right, so the logging gets written to a file on the SAS Server.
Mr Beer: Then over the page to 6.1 and 2.
Anne Chambers: Yes. So this is connecting to counters or central servers. Yes, I remember doing those things.
Mr Beer: Then down the page to 6.2.1.
Anne Chambers: Yes.
Mr Beer: Then over the page, please. Then scroll down to the bottom of the page. As I’d read this, Mrs Chambers, the system, if it had been carried into effect, required by this document, would have first required users to log in using their own names and passwords, then, when connecting to the counters, they logged in using a special user ID. That special user ID would be logged in real time against each and every action that the user then performed, yes?
Anne Chambers: Yes, I think that was the case, although I can’t recall – I don’t think I ever saw any of this logging. But yes, we did. But if we go back to what you were then asking me about for data changes on HNG-X, those weren’t made on the counter. That wasn’t – because the data was no longer held on the counters.
Mr Beer: So that explains the difference, does it, between what we read in Mr Parker’s minute of 2011 and what is described here?
Anne Chambers: Possibly, yes, because I think this was – there was a lot – all this background logging going on, which we were really almost meant to be unaware of. And I’m not sure if it did ever get checked to see, but there was this background audit of everything done via an SSH connection. Now, I can’t remember whether we – this implies that also we used SSH for connecting to the central systems. I’m not sure now if that included the Unix – the Oracle databases, which is what the 2010 document was talking about – changes to that.
So it may well be that if it was still via SSH, then that would have been an additional level of logging was going on automatically anyway but I really don’t know. This was not my area at all.
Mr Beer: I understand. Can we look at what you did understand as to the process, and look at your witness statement at page 57, please. In paragraph 199 at the foot of the page you say:
“I am asked whether a person using a branch terminal would be expressly notified by Horizon if changes were made using these access rights. It is hard to remember now what was done so long ago. If the branch had raised the service incident in the first place, via HSD, then I would probably have told them if I was going to make a change to their system to correct the problem. If they had not raised it, but instead we knew there was a problem because of a Reconciliation Report entry and the branch had not yet done their balance, I probably would not have contacted the branch, but would have informed my manager or the Fujitsu Problem Manager. If the problem was notified to the Post Office and they had authorised the change, I would expect Post Office to decide whether or not to notify the branch.”
So you say you would probably have notified a branch if you were going to make a change using remote access rights, if the postmaster had themselves initiated the incident.
Anne Chambers: Yes. As I said, it’s really hard to remember what would have been done, what was done in each case. I’ve seen various PEAKs in the preparation for this, some where I definitely did talk to the branch.
Mr Beer: And others where sometimes you didn’t?
Anne Chambers: And others where I didn’t, yes. Also, remember that, you know, making the changes would not necessarily be changes to financial data. One thing we did have to do sometimes was, where a branch was unable to use one of their counters because it had been left in a slightly inconsistent state, thinking it was still in the middle of balancing, and they could sort it out for themselves if the same user logged back on to that counter. But if that user had gone off on holiday for two weeks, that wasn’t very helpful. So there was a – by rewriting a – sorry, by updating an object in the message store, basically writing a new version of the object, we could clear that flag, which would then enable them to log on to the counter and proceed as normal.
That had absolutely no financial effect. So the number of changes where we were actually making a financial correction were very few and far between.
Mr Beer: You refer in this paragraph to your manager, ie in the cases where the problem had been picked up by a Reconciliation Report entry, you say you probably wouldn’t contact the branch but you’d have told your manager that you had made a change on the subpostmaster’s counter.
Anne Chambers: Yes.
Mr Beer: Are you referring there to Mr Peach and then Mr Parker?
Anne Chambers: Yes, it would have been. And if it was a change that had any financial implication, then it would have needed the – whatever the change control system at the time was, following.
Mr Beer: You say or you might have informed the Fujitsu Problem Manager. Who was the Fujitsu Problem Manager?
Anne Chambers: They varied over the years, but back for Legacy it was Mike Stewart and Mike Woolgar, I believe.
Mr Beer: Sorry, the second name?
Anne Chambers: Woolgar.
Mr Beer: Was it your understanding that any of those people, Mr Peach, Mr Parker, or either of the problem managers, would have contacted the branch to tell them that corrections had been made to their data?
Anne Chambers: No, I think it’s very unlikely that they would have contacted the branch.
Mr Beer: You say that in relation to – in the last part, that if the problem had been notified to the Post Office and they’d authorised the change, you would expect the Post Office to decide whether or not to notify the branch. What would affect whether or not the Post Office told the branch that their financial data had been altered?
Anne Chambers: I don’t know. I think that’s a question for Post Office.
Mr Beer: Did you know of some cases where the Post Office did not tell a subpostmaster that their financial data had been altered remotely by somebody within Fujitsu?
Anne Chambers: Yes, I think that definitely did happen.
Mr Beer: Did you follow the Group Litigation at all?
Anne Chambers: No, not until I realised that I was being mentioned quite a lot, which was very, very near the end of it.
Mr Beer: Did you read in that, did you see in that, that the Post Office was seeking to deny that the remote access that we are speaking about could occur?
Anne Chambers: I didn’t see that at the time, certainly, and I haven’t read up on it thoroughly.
Mr Beer: Sir, that’s an appropriate moment –
Sir Wyn Williams: Yes.
Mr Beer: – if it’s convenient to you.
Sir Wyn Williams: Certainly, yes. Can we take stock, so to speak? Are you still confident that all your questions and those of other legal representatives can be completed tomorrow?
Mr Beer: In respect of the first clause in that sentence, yes. The second clause, I’m in the hands of others. I’ve canvassed some views from some of my friends and I’m covering a lot of the ground that they’ve asked me to cover.
Sir Wyn Williams: Right. Well, would like you to discuss that briefly with your colleagues for this reason: that having said on Friday that we wouldn’t sit on Thursday and Friday of this week, I made an appointment for myself on Thursday morning which I do not now wish to give up, because they’re difficult to come by, and so I want that to be factored in to any arrangement, and I don’t want to inconvenience Mrs Chambers more than is strictly necessary.
So I’d be grateful if we could work out whether we will finish tomorrow and, if not, what we do.
Mr Beer: Sir, we will liaise, and I’ll report back to you overnight.
Sir Wyn Williams: Thank you.
(4.27 pm)
Sir Wyn Williams: (The hearing concluded until 10.00 am
the following day)